Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-32170

Improve the speculation for the inefficient tasks by the task metrics.

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.4.0
    • Scheduler, Spark Core
    • None

    Description

      1) Tasks will be speculated when meet certain conditions no matter they are inefficient or not,this would be a huge waste of cluster resources.
      2) In production,the speculation task comes from an efficient one will be killed finally,which is unnecessary and will waste of cluster resources.
      3) So, we should evaluate whether the task is inefficient by success tasks metrics firstly, and then decide to speculate it or not. The inefficient task will be speculated and efficient one will not, it is better for the cluster resources.

      Attachments

        Activity

          People

            weixiuli weixiuli
            weixiuli weixiuli
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: