Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-1703

option to balance load according to CPU available rather than slots available

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      To be able to accommodate for loading and running several different topologies on a cluster, one may want to assign more slots per node than there are CPUs available (many slots per CPU).

      For example, with 3 nodes with 3 CPUs each, defining 3 slots per node (9 slots total) limits the number of task that can be loaded at any one time.
      => Increasing the number of slots to 3 per CPU, gives 27 slots, allowing to load more topologies.

      In this configuration (3 nodes, 3 CPUs, 27 slots), if a job specifies a parallelism of 9, 9 slots out of 27 are used, but the distribution strategy gives preference to slots on the same host as the stream source to limit network IOs: as a consequence, the first job using 9 slots has the 9 slots allocated on a single host (the one on the same host as the stream source), even though this host only has 3 CPUs available.
      These 9 slots are therefore sharing the same 3 CPUs, while the other 2 nodes (6 CPUs) stay idle.

      The distribution strategy should take into account the number of CPUs available along with the number of slots rather than rely only on the slot availability.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                eleroy Emmanuel Leroy
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: