Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-13718

MemoryMonitor only uploads to GCS on restart

Details

    • Improvement
    • Status: Open
    • P3
    • Resolution: Unresolved
    • None
    • None
    • sdk-java-harness
    • None

    Description

      The current MemoryMonitor has an option to upload a profile to GCS. However it only uploads the profile to GCS when starting the memory monitor thread. This is currently done when the harness is started only.

      However MemoryMonitor's tryToDumpHeap is public and therefore can be called clients at any point (e.g. not only on worker restart). In this case we do not upload to GCS until the harness restarts which is not good end user behavior. It's currently called by StreamingDataflowWorker at a point of high memory but before the worker will actually crash, which is how we caught this behavior.

      Probably we should add a boolean to tryToHeapDump to get an idea for if the worker is about to die or if this is a pre-emptive dump. If it's pre-emptive we can probably immediately upload it to GCS as well if so configured.

      Attachments

        Activity

          People

            Unassigned Unassigned
            laraschmidt Lara Schmidt
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: