Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-4228

The FlinkRunner shouldn't require all of the values for a key to fit in memory

Details

    • New Feature
    • Status: Open
    • P3
    • Resolution: Unresolved
    • None
    • None
    • runner-flink
    • None

    Description

      The use of a reducer that adds all of the elements that it consumes to a list is the primary way in which this occurs - if instead, we produce a filtered iterable, or a collection of filtered iterables, we can lazily iterate over all of the contained elements without having to buffer all of the elements.

       

      For an example of where this occurs, see Concatenate in  FlinkBatchPortablePipelineTranslator.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              tgroh Thomas Groh
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated: