Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-6833

dataflow_metrics.py should be able to fully query the metrics

Details

    • New Feature
    • Status: Open
    • P3
    • Resolution: Unresolved
    • None
    • None
    • sdk-py-core
    • None

    Description

      dataflow_metrics.py expects a job object to be passed in, it cannot fully query the metrics on its own.

      Updating this to use the dataflow_client to also query the job itself (non trivial since the job passed in is a wrapper for the Job proto the dataflow_client obtains, this wrapper is created by dataflow_runner.) Though we may not need any of the extra wrapper features, since we only need access to the graph. But it is unclear if this will work, as the graph DatalowRunner generates is not the same graph as the one queried out.

       

      The job graph is use to translate step names

      Attachments

        Activity

          People

            Unassigned Unassigned
            ajamato@google.com Alex Amato

            Dates

              Created:
              Updated:

              Slack

                Issue deployment