Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-7684

WindmillStateCache grossly misunderestimates object size in its weighting function

Details

    • Bug
    • Status: Open
    • P3
    • Resolution: Unresolved
    • None
    • None
    • runner-dataflow
    • None

    Description

      As a little experiment, we set the WindmillStateCache size to 2 GB.  After we let a job run for a few hours, the workers all died from OOMs.  Digging into the memory usage, over 10 GB of heap was used by the WindmillStateCache.  The total weight the cache had calculated however was only ~800 MB.

      It looks like the StateCacheEntry calculation is way off.  The weight of an entry was calculated as only ~200, while the actual heap usage of it (as reported by YourKit at least) was around 2,400 bytes.

      Attachments

        Activity

          People

            Unassigned Unassigned
            SteveNiemitz Steve Niemitz
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated: