Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-7063

Stage Dataflow runner harness in Dataflow FnApi Python test suites that run using an unreleased SDK.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: sdk-py-core, test-failures
    • Labels:
      None

      Description

      Seems like this was the first post-commit failure: https://builds.apache.org/view/A-D/view/Beam/view/PostCommit/job/beam_PostCommit_Python_Verify/7855/

      (the one before this also failed but that looks like a flake)

       

       

      Looking at failed tests I noticed that some of the were due to failed Dataflow jobs. I noticed three issues.

      (1) Error in Dataflow jobs "This handler is only capable of dealing with urn:beam:sideinput:materialization:multimap:0.1 materializations but was asked to handle beam:side_input:multimap:v1 for PCollectionView with tag side0-FilterOutSpammers."

      https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-08_17_57_01-16397932959358575026?project=apache-beam-testing

      https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-08_18_07_11-1342017143600540529?project=apache-beam-testing

       

      (2)  BigQuery job failures (possibly this is transient)

      https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-08_18_25_22-16437560120793921616?project=apache-beam-testing

       

      (3) Some batch Dataflow jobs timed out and were cancelled (possibly also transient).

       

      (1) is probably due to https://github.com/apache/beam/commit/684f8130284a7c7979773300d04e5473ca0ac8f3 

       

      cc: Ahmet Altay Robert Burke Alan Myrvold

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tvalentyn Valentyn Tymofieiev
                Reporter:
                chamikara Chamikara Madhusanka Jayalath
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 5h 40m
                  5h 40m