Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-1251 Python 3 Support
  3. BEAM-6870

python 3 test_hourly_team_score_it fails with bigquery job id already exists

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: None
    • Fix Version/s: Not applicable
    • Component/s: sdk-py-core
    • Labels:
      None

      Description

      This IT test on python 3 fails with "Already exists job ..."

      14:42:03 ======================================================================
      14:42:03 ERROR: test_hourly_team_score_it (apache_beam.examples.complete.game.hourly_team_score_it_test.HourlyTeamScoreIT)
      14:42:03 ----------------------------------------------------------------------
      14:42:03 Traceback (most recent call last):
      14:42:03   File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify_PR/src/sdks/python/apache_beam/examples/complete/game/hourly_team_score_it_test.py", line 89, in test_hourly_team_score_it
      14:42:03     self.test_pipeline.get_full_options_as_args(**extra_opts))
      14:42:03   File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify_PR/src/sdks/python/apache_beam/examples/complete/game/hourly_team_score.py", line 295, in run
      14:42:03     }, options.view_as(GoogleCloudOptions).project))
      14:42:03   File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify_PR/src/sdks/python/apache_beam/pipeline.py", line 426, in __exit__
      14:42:03     self.run().wait_until_finish()
      14:42:03   File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify_PR/src/sdks/python/apache_beam/pipeline.py", line 406, in run
      14:42:03     self._options).run(False)
      14:42:03   File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify_PR/src/sdks/python/apache_beam/pipeline.py", line 419, in run
      14:42:03     return self.runner.run_pipeline(self, self._options)
      14:42:03   File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify_PR/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py", line 64, in run_pipeline
      14:42:03     self.result.wait_until_finish(duration=wait_duration)
      14:42:03   File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify_PR/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py", line 1217, in wait_until_finish
      14:42:03     (self.state, getattr(self._runner, 'last_error_msg', None)), self)
      14:42:03 apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error:
      14:42:03 Traceback (most recent call last):
      14:42:03   File "apache_beam/runners/common.py", line 732, in apache_beam.runners.common.DoFnRunner.process
      14:42:03   File "apache_beam/runners/common.py", line 555, in apache_beam.runners.common.PerWindowInvoker.invoke_process
      14:42:03   File "apache_beam/runners/common.py", line 624, in apache_beam.runners.common.PerWindowInvoker._invoke_per_window
      14:42:03   File "apache_beam/runners/common.py", line 816, in apache_beam.runners.common._OutputProcessor.process_outputs
      14:42:03   File "apache_beam/runners/common.py", line 831, in apache_beam.runners.common._OutputProcessor.process_outputs
      14:42:03   File "/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/bigquery_file_loads.py", line 385, in process
      14:42:03     create_disposition=self.create_disposition)
      14:42:03   File "/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/bigquery_tools.py", line 560, in perform_load_job
      14:42:03     write_disposition=write_disposition)
      14:42:03   File "/usr/local/lib/python3.5/site-packages/apache_beam/utils/retry.py", line 195, in wrapper
      14:42:03     return fun(*args, **kwargs)
      14:42:03   File "/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/bigquery_tools.py", line 327, in _insert_load_job
      14:42:03     response = self.client.jobs.Insert(request)
      14:42:03   File "/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py", line 342, in Insert
      14:42:03     upload=upload, upload_config=upload_config)
      14:42:03   File "/usr/local/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 731, in _RunMethod
      14:42:03     return self.ProcessHttpResponse(method_config, http_response, request)
      14:42:03   File "/usr/local/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 737, in ProcessHttpResponse
      14:42:03     self.__ProcessHttpResponse(method_config, http_response, request))
      14:42:03   File "/usr/local/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 604, in __ProcessHttpResponse
      14:42:03     http_response, method_config=method_config, request=request)
      14:42:03 apitools.base.py.exceptions.HttpConflictError: HttpError accessing <https://www.googleapis.com/bigquery/v2/projects/apache-beam-testing/jobs?alt=json>: response: <{'status': '409', 'content-type': 'application/json; charset=UTF-8', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'cache-control': 'private, max-age=0', '-content-encoding': 'gzip', 'date': 'Wed, 20 Mar 2019 13:29:44 GMT', 'server': 'GSE', 'content-length': '364', 'x-xss-protection': '1; mode=block', 'vary': 'Origin, X-Origin', 'x-frame-options': 'SAMEORIGIN', 'expires': 'Wed, 20 Mar 2019 13:29:44 GMT'}>, content <{
      14:42:03  "error": {
      14:42:03   "errors": [
      14:42:03    {
      14:42:03     "domain": "global",
      14:42:03     "reason": "duplicate",
      14:42:03     "message": "Already Exists: Job apache-beam-testing:US.beam_load_2019_03_20_132147_71_1fea0de5d8e357a03e39484b3ebe0123_0"
      14:42:03    }
      14:42:03   ],
      14:42:03   "code": 409,
      14:42:03   "message": "Already Exists: Job apache-beam-testing:US.beam_load_2019_03_20_132147_71_1fea0de5d8e357a03e39484b3ebe0123_0"
      14:42:03  }
      14:42:03 }
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              Juta Juta Staes
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: