Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-65

SplittableDoFn

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • P3
    • Resolution: Fixed
    • None
    • 2.2.0
    • beam-model
    • None

    Description

      SplittableDoFn is a proposed enhancement for "dynamically splittable work" to the Beam model.

      Among other things, it would allow a unified implementation of bounded/unbounded sources with dynamic work rebalancing and the ability to express multiple scalable steps (e.g., global expansion -> file sizing & parsing -> splitting files into independently-processable blocks) via composition rather than inheritance.

      This would make it much easier to implement many types of sources, to modify and reuse existing sources. Also, it would improve scalability of the Beam model by moving things like splitting a source from the control plane (where it is today – glob -> List<FileBasedSource> sent over service APIs) into the data plane (PCollection<Glob> -> PCollection<FileName> -> ...).

      Attachments

        Issue Links

          incorporates

          Bug - A problem which impairs or prevents the functions of the product. BEAM-1903 Splittable DoFn should report watermarks via ProcessContext

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. BEAM-1904 Remove DoFn.ProcessContinuation

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. BEAM-2301 Standard expansion of SDF should be in runners-core-construction

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. BEAM-1377 Support Splittable DoFn in Dataflow streaming runner

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. BEAM-1855 Support Splittable DoFn in Flink Streaming runner

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved
          is depended upon by

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-3301 Go SplittableDoFn support

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved
          is related to

          New Feature - A new feature of the product, which has yet to be developed. BEAM-644 Primitive to shift the watermark while assigning timestamps

          • P3 - Non-urgent bugs, features, and improvements
          • Open
          links to

          Web Link GitHub Pull Request #896

          Web Link GitHub Pull Request #1824

          Web Link GitHub Pull Request #1895

          Web Link GitHub Pull Request #2462

          Activity

            People

              jkff Eugene Kirpichov
              dhalperi Dan Halperin
              Votes:
              3 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: