Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-10240

Pluggable scheduling strategy for batch jobs

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Later
    • 1.7.0
    • None
    • Runtime / Coordination

    Description

      Currently batch jobs are scheduled with LAZY_FROM_SOURCES strategy: source tasks are scheduled in the beginning, and other tasks are scheduled once their input data is consumable.

      However, input data consumable does not always mean the task can work at once. 

       

      One example is the hash join operation, where the operator first consumes one side(we call it build side) to setup a table, then consumes the other side(we call it probe side) to do the real join work. If the probe side is started early, it just get stuck on back pressure as the join operator will not consume data from it before the building stage is done, causing a waste of resources.

      If we have the probe side task started after the build stage is done, both the build and probe side can have more computing resources as they are staggered.

       

      That's why we think a flexible scheduling strategy is needed, allowing job owners to customize the vertex schedule order and constraints. Better resource utilization usually means better performance.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              zhuzh Zhu Zhu
              Votes:
              1 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: