Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-7045

Element counters in the Web UI graph representations for transforms for Python streaming jobs in Google Cloud Dataflow

    XMLWordPrintableJSON

Details

    • Important
    • Hide
      Unsupported features
      Python streaming execution does not currently support the following features.

      General Beam features
      These unsupported Beam features apply to all runners.

      State and Timers APIs
      Custom source API
      Splittable DoFn API
      Handling of late data
      User-defined custom WindowFn
      DataflowRunner specific features
      Additionally, DataflowRunner does not currently support the following Cloud Dataflow specific features with Python streaming execution.

      Streaming autoscaling
      Updating existing pipelines
      Cloud Dataflow Templates
      Some monitoring features, such as msec counters, display data, metrics, and element counts for transforms. However, logging, watermarks, and element counts for sources are supported.
      Show
      Unsupported features Python streaming execution does not currently support the following features. General Beam features These unsupported Beam features apply to all runners. State and Timers APIs Custom source API Splittable DoFn API Handling of late data User-defined custom WindowFn DataflowRunner specific features Additionally, DataflowRunner does not currently support the following Cloud Dataflow specific features with Python streaming execution. Streaming autoscaling Updating existing pipelines Cloud Dataflow Templates Some monitoring features, such as msec counters, display data, metrics, and element counts for transforms. However, logging, watermarks, and element counts for sources are supported.

    Description

      Users don't see the element counters in transforms in the Web UI graph representation when running a Python streaming job, which is expected behavior according to this Beam page.

      The feature request is to enable the element counters in the Web UI graph representations for transforms for Python streaming jobs in Google Cloud Dataflow.

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. BEAM-7969 Streaming Dataflow worker doesn't report FnAPI metrics.

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              efimmuratov Fim
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: