Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-14447

BigQueryWriteIntegrationTests.test_big_query_write_insert_errors_reporting failing in Python PostCommit

Details

    Description

      A new test added in https://github.com/apache/beam/pull/17517 is failing at HEAD.

       

      https://ci-beam.apache.org/job/beam_PostCommit_Python37/5198/testReport/junit/apache_beam.io.gcp.bigquery_write_it_test/BigQueryWriteIntegrationTests/test_big_query_write_insert_errors_reporting/

       

              pipeline_verifiers = [
                  BigqueryFullResultMatcher(
                      project=self.project,
                      query="SELECT number, str FROM %s" % table_id,
                      data=[(1, 'some_string')]),
                  BigqueryFullResultMatcher(
                      project=self.project,
                      query="SELECT table, reason, row_json FROM %s" % errors_table_id,
                      data=
                      [(
                          table_id,
                          '[{"reason": "invalid", "location": "", "debugInfo": "", \
          "message": "Missing required field: Msg_0_CLOUD_QUERY_TABLE.str."}]',
                          '{"number": 2}'),
                       (
                           table_id,
                           '[{"reason": "invalid", "location": "additional_field_str", \
          "debugInfo": "", "message": "no such field: additional_field_str."}]',
                           '{"number": 3, "str": "some_string", "additional_field_str": \
          "some_string"}')])
              ]
          
              args = self.test_pipeline.get_full_options_as_args(
                  on_success_matcher=hc.all_of(*pipeline_verifiers))
          
              with beam.Pipeline(argv=args) as p:
                # pylint: disable=expression-not-assigned
                errors = (
                    p | 'create' >> beam.Create(input_data)
                    | 'write' >> beam.io.WriteToBigQuery(
                        table_id,
                        schema=table_schema,
                        create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
                        write_disposition=beam.io.BigQueryDisposition.WRITE_EMPTY))
                (
                    errors["FailedRows"]
                    | 'ParseErrors' >> beam.Map(
                        lambda err: {
                            "table": err[0],
                            "reason": json.dumps(err[2]),
                            "row_json": json.dumps(err[1])
                        })
      >             | 'WriteErrors' >> beam.io.WriteToBigQuery(
                        errors_table_id,
                        schema=errors_table_schema,
                        create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
                        write_disposition=beam.io.BigQueryDisposition.WRITE_EMPTY))
      E                 KeyError: 'FailedRows'
      
      apache_beam/io/gcp/bigquery_write_it_test.py:461: KeyError 

      Attachments

        Issue Links

          Activity

            People

              pabloem Pablo Estrada
              bhulette Brian Hulette
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 50m
                  50m