Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-1251 Python 3 Support
  3. BEAM-5785

Validates runner tests fail with: Cannot convert bytes value to JSON value

Details

    • Sub-task
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • None
    • Not applicable
    • runner-dataflow
    • None

    Description

      I run a random ValidatesRunner test on Python 3 with TestsDataflowRunner. The test failed before submitting job to the service.

      More details about my env and test:

      Python version: 3.5.3
      Test: apache_beam.transforms.ptransform_test:PTransformTest.test_multiple_empty_outputs
      Command:

      python setup.py nosetests \
        --tests apache_beam.transforms.ptransform_test:PTransformTest.test_multiple_empty_outputs  \
        --nocapture \
        --test-pipeline-options=" \                             
              --runner=TestDataflowRunner \
              --project=<my_project> \
              --staging_location=<my_staging> \
              --temp_location=<my_temp> \
              --output=<my_output> \                                                                      
              --sdk_location=.../beam/sdks/python/dist/apache-beam-2.9.0.dev0.tar.gz \
              --num_workers=1"
      

      Here are the stacktrace from my console:

      ======================================================================
      ERROR: test_multiple_empty_outputs (apache_beam.transforms.ptransform_test.PTransformTest)
      ----------------------------------------------------------------------
      Traceback (most recent call last):
        File ".../beam/sdks/python/apache_beam/transforms/ptransform_test.py", line 284, in test_multiple_empty_outputs
          pipeline.run()
        File ".../beam/sdks/python/apache_beam/testing/test_pipeline.py", line 107, in run
          else test_runner_api))
        File ".../beam/sdks/python/apache_beam/pipeline.py", line 403, in run
          self.to_runner_api(), self.runner, self._options).run(False)
        File ".../beam/sdks/python/apache_beam/pipeline.py", line 416, in run
          return self.runner.run_pipeline(self)
        File ".../beam/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py", line 50, in run_pipeline
          self.result = super(TestDataflowRunner, self).run_pipeline(pipeline)
        File ".../beam/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py", line 374, in run_pipeline
          super(DataflowRunner, self).run_pipeline(pipeline)
        File ".../beam/sdks/python/apache_beam/runners/runner.py", line 176, in run_pipeline
          pipeline.visit(RunVisitor(self))
        File ".../beam/sdks/python/apache_beam/pipeline.py", line 444, in visit
          self._root_transform().visit(visitor, self, visited)
        File ".../beam/sdks/python/apache_beam/pipeline.py", line 780, in visit
          part.visit(visitor, pipeline, visited)
        File ".../beam/sdks/python/apache_beam/pipeline.py", line 780, in visit
          part.visit(visitor, pipeline, visited)
        File ".../beam/sdks/python/apache_beam/pipeline.py", line 783, in visit
          visitor.visit_transform(self)
        File ".../beam/sdks/python/apache_beam/runners/runner.py", line 171, in visit_transform
          self.runner.run_transform(transform_node)
        File ".../beam/sdks/python/apache_beam/runners/runner.py", line 214, in run_transform
          return m(transform_node)
        File ".../beam/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py", line 846, in run_Read
          source_dict)
        File ".../beam/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py", line 85, in add_property
          key=name, value=to_json_value(value, with_type=with_type)))
        File ".../beam/sdks/python/apache_beam/internal/gcp/json_value.py", line 104, in to_json_value
          key=k, value=to_json_value(v, with_type=with_type)))
        File ".../beam/sdks/python/apache_beam/internal/gcp/json_value.py", line 104, in to_json_value
          key=k, value=to_json_value(v, with_type=with_type)))
        File ".../beam/sdks/python/apache_beam/internal/gcp/json_value.py", line 124, in to_json_value
          raise TypeError('Cannot convert %s to a JSON value.' % repr(obj))
      TypeError: Cannot convert b'eNpVjrEKwjAURRNjq6ZOfkVd8hOZxFGQLhJe0ycW2saXpA5CQf/ctro4Xc7hwr0vYeEO9oamRGhV9NCFq/NtUNYjRDTB9d6iNHrG05eI7d/EB1rkRcYYM9FFaEyon0jiKIrd5AL6GppRVeYBTY+BlhdKcs05pZoLWmme0BqLdCpbV6Gnzd+X2YVfyDP4Qxf1BJLkOJ8NtC37Un0AfYQ+kQ==' to a JSON value.
      -------------------- >> begin captured logging << --------------------
      avro.schema: Level 5: Register new name for 'org.apache.avro.file.Header'
      avro.schema: Level 5: Register new name for 'org.apache.avro.file.magic'
      avro.schema: Level 5: Register new name for 'org.apache.avro.file.sync'
      avro.schema: Level 5: Register new name for 'example.avro.User'
      root: WARNING: snappy is not installed; some tests will be skipped.
      avro.schema: Level 5: Register new name for 'example.avro.User'
      root: WARNING: Tensorflow is not installed, so skipping some tests.
      root: ERROR: Error while visiting Some Numbers/Read
      --------------------- >> end captured logging << ---------------------
      Ran 1 test in 1.319s
      

      How to reproduce:
      1. create virtualenv with python 3
      2. export BEAM_EXPERIMENTAL_PY3=1 to env path
      3. install gcp,test dependencies
      4. build python tar file
      5. run test

      Attachments

        Activity

          People

            markflyhigh Mark Liu
            markflyhigh Mark Liu
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: