Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-5859

Improve Traceability of Pipeline translation

Details

    Description

      Users often ask how they can reason about the pipeline translation. The Flink UI display a confusingly large graph without any trace of the original Beam pipeline:

      WordCount:

      TFX:

      Some aspects which make understanding these graphs hard:

      • Users don't know how the Runner maps Beam to Flink concepts
      • The UI is awfully slow / hangs when the pipeline is reasonable complex
      • The operator names seem to use transform.getUniqueName() which doesn't generate readable name
      • So called Chaining combines operators into a single operator which makes understanding which Beam concept belongs to which Flink concept even harder

       

      Attachments

        1. wordcount.png
          62 kB
          Maximilian Michels
        2. tfx.png
          252 kB
          Maximilian Michels

        Activity

          People

            Unassigned Unassigned
            mxm Maximilian Michels
            Votes:
            1 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 4h
                4h