Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-12335

Apply basic fusion to Java DirectRunner to avoid keeping all intermittent results in memory

Details

    • Improvement
    • Status: Open
    • P3
    • Resolution: Unresolved
    • None
    • None
    • runner-direct
    • None

    Description

      Current java direct runner doesn't fuse transforms into steps. Instead, it almost executes each transform one by one. It results in memory pressure when any transform is high-fanout.

      We already have a simple fusion logic in Java SDK(https://github.com/apache/beam/blob/master/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPipelineFuser.java). Work remaining here might be:

      • Apply such fusion into DirectRunner
      • Change the DirectRunner to be able run the fused steps.

      I understand that DirectRunner doesn't expect processing large volume data and changing DirectRunner execution might be a fair amount of work.

      Attachments

        Activity

          People

            Unassigned Unassigned
            boyuanz Boyuan Zhang
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated: