Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-8608

Chain DoFns in Flink batch runner when possible.

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • 2.16.0
    • 2.23.0
    • runner-flink
    • None

    Description

      Right now, in Batch runner, DoFn is executed using MapPartition operator (FlinkDoFnFunction), which doesn't have chained driver implementation.

      We need to reimplement DoFnFunction with FlatMap to allow chaining.

      Attached is the execution graph for the same pipeline, before and after the patch.

      Attachments

        1. Screen Shot 2019-11-07 at 10.35.22.png
          283 kB
          David Morávek
        2. Screen Shot 2019-11-07 at 10.35.07.png
          489 kB
          David Morávek

        Issue Links

          depends upon

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-14709 Allow outputting elements in close method of chained drivers.

          • Major - Major loss of function.
          • Closed
          is depended upon by

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-8848 Flink: Memory efficient GBK implementation for batch runner

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved
          links to

          Web Link GitHub Pull Request #11858

          Activity

            People

              dmvk David Morávek
              dmvk David Morávek
              Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 10m
                  1h 10m