Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24677

TaskSetManager not updating successfulTaskDurations for old stage attempts

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 2.3.1
    • 2.2.3, 2.3.3, 2.4.0
    • Spark Core
    • None

    Description

      When introducing SPARK-23433 , maybe cause stop sparkcontext.

      ERROR Utils: uncaught error in thread task-scheduler-speculation, stopping SparkContext
      java.util.NoSuchElementException: MedianHeap is empty.
      at org.apache.spark.util.collection.MedianHeap.median(MedianHeap.scala:83)
      at org.apache.spark.scheduler.TaskSetManager.checkSpeculatableTasks(TaskSetManager.scala:968)
      at org.apache.spark.scheduler.Pool$$anonfun$checkSpeculatableTasks$1.apply(Pool.scala:94)
      at org.apache.spark.scheduler.Pool$$anonfun$checkSpeculatableTasks$1.apply(Pool.scala:93)
      at scala.collection.Iterator$class.foreach(Iterator.scala:742)
      at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
      at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
      at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
      at org.apache.spark.scheduler.Pool.checkSpeculatableTasks(Pool.scala:93)
      at org.apache.spark.scheduler.Pool$$anonfun$checkSpeculatableTasks$1.apply(Pool.scala:94)
      at org.apache.spark.scheduler.Pool$$anonfun$checkSpeculatableTasks$1.apply(Pool.scala:93)
      

      Attachments

        Activity

          People

            dzcxzl dzcxzl
            dzcxzl dzcxzl
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: