Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-6584

Python SDK creates job graphs with duplicated states when using fn_api execution mode.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: sdk-py-harness
    • Labels:
      None

      Description

      We observed this on apache_beam.examples.wordcount with Dataflow runner.

      The graph for this wordcount job contains two steps with the same name "write/Write/WriteImpl/FinalizeWrite/MapToVoidKey1".

      ...
       {
              "kind": "PAR_DO_KIND",
              "id": "s41",
              "name": "write/Write/WriteImpl/FinalizeWrite/MapToVoidKey1",
              "displayData": [
                {
                  "key": "fn",
                  "namespace": "apache_beam.transforms.core.ParDo",
                  "strValue": "apache_beam.transforms.core.CallableWrapperDoFn",
                  "shortStrValue": "CallableWrapperDoFn",
                  "label": "Transform Function"
                },
                {
                  "key": "fn",
                  "namespace": "apache_beam.transforms.core.CallableWrapperDoFn",
                  "strValue": "\u003clambda\u003e",
                  "label": "Transform Function"
                }
              ],
              "outputCollectionName": [
                "write/Write/WriteImpl/FinalizeWrite/MapToVoidKey1.out0"
              ],
              "inputCollectionName": [
                "write/Write/WriteImpl/Extract.out0"
              ]
            },
      ...
      {
              "kind": "PAR_DO_KIND",
              "id": "s31",
              "name": "write/Write/WriteImpl/FinalizeWrite/MapToVoidKey1",
              "displayData": [
                {
                  "key": "fn",
                  "namespace": "apache_beam.transforms.core.ParDo",
                  "strValue": "apache_beam.transforms.core.CallableWrapperDoFn",
                  "shortStrValue": "CallableWrapperDoFn",
                  "label": "Transform Function"
                },
                {
                  "key": "fn",
                  "namespace": "apache_beam.transforms.core.CallableWrapperDoFn",
                  "strValue": "\u003clambda\u003e",
                  "label": "Transform Function"
                }
              ],
              "outputCollectionName": [
                "write/Write/WriteImpl/FinalizeWrite/MapToVoidKey1.out0"
              ],
              "inputCollectionName": [
                "write/Write/WriteImpl/Extract.out0"
              ]
            },
      ...
      
      

      CC: Andrea Foegler Ahmet Altay Robert Bradshaw

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              tvalentyn Valentyn Tymofieiev
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: