Details
-
Sub-task
-
Status: Resolved
-
P2
-
Resolution: Fixed
-
None
-
None
Description
Currently, we collect time (distribution) and bytes (counter) metrics from one ParDo (called MetricsMonitor) that is put in pipelines in one, arbitrarily chosen place (usually "in the middle" of pipeline's graph. In some cases, invalid time (or total bytes count) is registered.
Taking this discussion into account, ideally, we'd like to:
- collect runtime by recording time at the root and sink(s) of the pipeline
- collect total bytes in a separate ParDo that allows deciding what byte amount do we actually want to collect (now it's coupled to the time-collecting Monitor which is inconvenient).
Attachments
Issue Links
- links to