Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26275

Flaky test: pyspark.mllib.tests.test_streaming_algorithms StreamingLogisticRegressionWithSGDTests.test_training_and_prediction

    XMLWordPrintableJSON

    Details

    • Type: Test
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: MLlib, PySpark
    • Labels:
      None

      Description

      Looks this test is flaky

      https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99704/console
      https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99569/console
      https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99644/console
      https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99548/console
      https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99454/console
      https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99609/console

      ======================================================================
      FAIL: test_training_and_prediction (pyspark.mllib.tests.test_streaming_algorithms.StreamingLogisticRegressionWithSGDTests)
      Test that the model improves on toy data with no. of batches
      ----------------------------------------------------------------------
      Traceback (most recent call last):
        File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/mllib/tests/test_streaming_algorithms.py", line 367, in test_training_and_prediction
          self._eventually(condition)
        File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/mllib/tests/test_streaming_algorithms.py", line 78, in _eventually
          % (timeout, lastValue))
      AssertionError: Test failed due to timeout after 30 sec, with last condition returning: Latest errors: 0.67, 0.71, 0.78, 0.7, 0.75, 0.74, 0.73, 0.69, 0.62, 0.71, 0.69, 0.75, 0.72, 0.77, 0.71, 0.74
      
      ----------------------------------------------------------------------
      Ran 13 tests in 185.051s
      
      FAILED (failures=1, skipped=1)
      

      This looks happening after increasing the parallelism in Jenkins to speed up. I am able to reproduce this manually when the resource usage is heavy with manual decrease of timeout.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                hyukjin.kwon Hyukjin Kwon
                Reporter:
                hyukjin.kwon Hyukjin Kwon
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: