Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-5859

Improve Traceability of Pipeline translation

    XMLWordPrintableJSON

    Details

      Description

      Users often ask how they can reason about the pipeline translation. The Flink UI display a confusingly large graph without any trace of the original Beam pipeline:

      WordCount:

      TFX:

      Some aspects which make understanding these graphs hard:

      • Users don't know how the Runner maps Beam to Flink concepts
      • The UI is awfully slow / hangs when the pipeline is reasonable complex
      • The operator names seem to use transform.getUniqueName() which doesn't generate readable name
      • So called Chaining combines operators into a single operator which makes understanding which Beam concept belongs to which Flink concept even harder

       

        Attachments

        1. tfx.png
          252 kB
          Maximilian Michels
        2. wordcount.png
          62 kB
          Maximilian Michels

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              mxm Maximilian Michels
            • Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 4h
                4h