Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-7063

Stage Dataflow runner harness in Dataflow FnApi Python test suites that run using an unreleased SDK.

Details

    Description

      Seems like this was the first post-commit failure: https://builds.apache.org/view/A-D/view/Beam/view/PostCommit/job/beam_PostCommit_Python_Verify/7855/

      (the one before this also failed but that looks like a flake)

       

       

      Looking at failed tests I noticed that some of the were due to failed Dataflow jobs. I noticed three issues.

      (1) Error in Dataflow jobs "This handler is only capable of dealing with urn:beam:sideinput:materialization:multimap:0.1 materializations but was asked to handle beam:side_input:multimap:v1 for PCollectionView with tag side0-FilterOutSpammers."

      https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-08_17_57_01-16397932959358575026?project=apache-beam-testing

      https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-08_18_07_11-1342017143600540529?project=apache-beam-testing

       

      (2)  BigQuery job failures (possibly this is transient)

      https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-04-08_18_25_22-16437560120793921616?project=apache-beam-testing

       

      (3) Some batch Dataflow jobs timed out and were cancelled (possibly also transient).

       

      (1) is probably due to https://github.com/apache/beam/commit/684f8130284a7c7979773300d04e5473ca0ac8f3 

       

      cc: altay lostluck alanmyrvold

      Attachments

        Issue Links

          Activity

            People

              tvalentyn Valentyn Tymofieiev
              chamikara Chamikara Madhusanka Jayalath
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 5h 40m
                  5h 40m