Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25780

Scheduling the tasks which have no higher level locality first

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: In Progress
    • Minor
    • Resolution: Unresolved
    • 3.1.0
    • None
    • Scheduler, Spark Core
    • None

    Description

      For example:
      An application has two executors: (exe1, host1), (exe2,host2).
      And 3 tasks with locality: {task0, Seq(TaskLocation("host1", "exec1"))}, {task1, Seq(TaskLocation("host1", "exec1"), TaskLocation("host2"))},  {task2, Seq(TaskLocation("host2")}
      If task0 is runing in exe1, when `allowedLocality` is NODE_LOCAL for exe2, it is better to schedule task2 fisrt, not task1, because task1 may be scheduled to exe1 later.

      Attachments

        Activity

          People

            Unassigned Unassigned
            10110346 liuxian
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: