Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-5980 Add load tests for Core Apache Beam operations
  3. BEAM-6100

Runtime and total bytes metrics are not collected properly

Details

    • Sub-task
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • None
    • Not applicable
    • testing
    • None

    Description

      Currently, we collect time (distribution) and bytes (counter) metrics from one ParDo (called MetricsMonitor) that is put in pipelines in one, arbitrarily chosen place (usually "in the middle" of pipeline's graph. In some cases, invalid time (or total bytes count) is registered. 

      Taking this discussion into account, ideally, we'd like to:

      • collect runtime by recording time at the root and sink(s) of the pipeline
      • collect total bytes in a separate ParDo that allows deciding what byte amount do we actually want to collect (now it's coupled to the time-collecting Monitor which is inconvenient).

      Attachments

        Issue Links

          Activity

            People

              ŁukaszG Lukasz Gajowy
              ŁukaszG Lukasz Gajowy
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 4.5h
                  4.5h