Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24417 Build and Run Spark on JDK11
  3. SPARK-28735

MultilayerPerceptronClassifierTest.test_raw_and_probability_prediction fails on JDK11

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: ML, PySpark
    • Labels:
      None

      Description

      Build Spark and run PySpark UT with JDK11. The last commented `assertTrue` failed.

      $ build/sbt -Phadoop-3.2 test:package
      $ python/run-tests --testnames 'pyspark.ml.tests.test_algorithms' --python-executables python
      ...
      ======================================================================
      FAIL: test_raw_and_probability_prediction (pyspark.ml.tests.test_algorithms.MultilayerPerceptronClassifierTest)
      ----------------------------------------------------------------------
      Traceback (most recent call last):
        File "/Users/dongjoon/APACHE/spark-master/python/pyspark/ml/tests/test_algorithms.py", line 89, in test_raw_and_probability_prediction
          self.assertTrue(np.allclose(result.rawPrediction, expected_rawPrediction, atol=1E-4))
      AssertionError: False is not true
      
      class MultilayerPerceptronClassifierTest(SparkSessionTestCase):
          def test_raw_and_probability_prediction(self):
              data_path = "data/mllib/sample_multiclass_classification_data.txt"
              df = self.spark.read.format("libsvm").load(data_path)
              mlp = MultilayerPerceptronClassifier(maxIter=100, layers=[4, 5, 4, 3],
                                                   blockSize=128, seed=123)
              model = mlp.fit(df)
              test = self.sc.parallelize([Row(features=Vectors.dense(0.1, 0.1, 0.25, 0.25))]).toDF()
              result = model.transform(test).head()
              expected_prediction = 2.0
              expected_probability = [0.0, 0.0, 1.0]
      	        expected_rawPrediction = [-11.6081922998, -8.15827998691, 22.17757045]
      	        self.assertTrue(result.prediction, expected_prediction)
      	        self.assertTrue(np.allclose(result.probability, expected_probability, atol=1E-4))
      	        self.assertTrue(np.allclose(result.rawPrediction, expected_rawPrediction, atol=1E-4))
      	        # self.assertTrue(np.allclose(result.rawPrediction, expected_rawPrediction, atol=1E-4))
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                hyukjin.kwon Hyukjin Kwon
                Reporter:
                dongjoon Dongjoon Hyun
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: