Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-5713

Flink portable runner schedules all tasks of streaming job on same task manager

    XMLWordPrintableJSON

Details

    Description

      The cluster has 9 task managers and 144 task slots total. A simple streaming pipeline with parallelism of 8 will get all tasks scheduled on the same task manager, causing the host to be fully booked and the remaining cluster idle.

      Attachments

        1. With RichParallelSourceFunction and parallelism 5.png
          5 kB
          Maximilian Michels
        2. Different SlotSharingGroup.png
          4 kB
          Maximilian Michels
        3. image-2018-10-11-11-43-50-333.png
          221 kB
          Thomas Weise
        4. image-2018-10-11-16-20-45-221.png
          176 kB
          Thomas Weise

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-1003 Spread out scheduling strategy

          • Major - Major loss of function.
          • Closed
          links to

          Web Link GitHub Pull Request #6654

          Activity

            People

              mxm Maximilian Michels
              thw Thomas Weise
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 20m
                  1h 20m