Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-8212

StatefulDoFnRunner creates GC timers for every record

Details

    • Bug
    • Status: Resolved
    • P3
    • Resolution: Duplicate
    • None
    • Not applicable
    • runner-core
    • None

    Description

      Hi 

      So currently the StatefulParDoFn create timers for all the records.
      https://github.com/apache/beam/blob/master/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java#L211

      This becomes a problem if you are using GlobalWindows for streaming where these timers get created and never get closed since the window will never close.

      This is a problem especially if your memory bound in rocksDB where these timers take up potential space and sloe the pipelines considerably.

      Was wondering that if the pipeline runs in global windows we should avoid adding timers to it at all?

       

       

       

      Attachments

        Issue Links

          Activity

            People

              kenn Kenneth Knowles
              aiyangar Akshay Iyangar
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: