Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26275

Flaky test: pyspark.mllib.tests.test_streaming_algorithms StreamingLogisticRegressionWithSGDTests.test_training_and_prediction

    XMLWordPrintableJSON

Details

    • Test
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • MLlib, PySpark
    • None

    Description

      Looks this test is flaky

      https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99704/console
      https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99569/console
      https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99644/console
      https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99548/console
      https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99454/console
      https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99609/console

      ======================================================================
      FAIL: test_training_and_prediction (pyspark.mllib.tests.test_streaming_algorithms.StreamingLogisticRegressionWithSGDTests)
      Test that the model improves on toy data with no. of batches
      ----------------------------------------------------------------------
      Traceback (most recent call last):
        File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/mllib/tests/test_streaming_algorithms.py", line 367, in test_training_and_prediction
          self._eventually(condition)
        File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/mllib/tests/test_streaming_algorithms.py", line 78, in _eventually
          % (timeout, lastValue))
      AssertionError: Test failed due to timeout after 30 sec, with last condition returning: Latest errors: 0.67, 0.71, 0.78, 0.7, 0.75, 0.74, 0.73, 0.69, 0.62, 0.71, 0.69, 0.75, 0.72, 0.77, 0.71, 0.74
      
      ----------------------------------------------------------------------
      Ran 13 tests in 185.051s
      
      FAILED (failures=1, skipped=1)
      

      This looks happening after increasing the parallelism in Jenkins to speed up. I am able to reproduce this manually when the resource usage is heavy with manual decrease of timeout.

      Attachments

        Issue Links

          Activity

            People

              gurwls223 Hyukjin Kwon
              gurwls223 Hyukjin Kwon
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: