Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-1251 Python 3 Support
  3. BEAM-5788

wordcount_fnapi_it failed on TestDataflowRunner because of JSON string decoding error

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.10.0
    • Component/s: test-failures
    • Labels:
      None

      Description

      Similar to BEAM-5785, wordcount_fnapi_it failed on Python 3 when running with TestDataflowRunner. Got TypeError: the JSON object must be str, not 'bytes'. This error cause infinite retry before job could submitted to service.

      More details about my env and test:
      Python version: 3.5.3
      Test: apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_fnapi_it
      Command:

      python setup.py nosetests \
        --tests apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_fnapi_it  \
        --nocapture \
        --nologcapture \
        --test-pipeline-options=" \                             
              --runner=TestDataflowRunner \
              --project=<my_project> \
              --staging_location=<my_staging> \
              --temp_location=<my_temp> \
              --output=<my_output> \                                                                      
              --sdk_location=.../beam/sdks/python/dist/apache-beam-2.9.0.dev0.tar.gz \
              --num_workers=1"
      

      Stacktrace:

      WARNING:root:Retry with exponential backoff: waiting for 7.661876827680761 seconds before retrying exists because we caught exception: TypeError: the JSON object must be str, not 'bytes'
       Traceback for above exception (most recent call last):
        File ".../beam/sdks/python/apache_beam/utils/retry.py", line 184, in wrapper
          return fun(*args, **kwargs)
        File ".../beam/sdks/python/apache_beam/io/gcp/gcsio.py", line 375, in exists
          self.client.objects.Get(request)  # metadata
        File ".../beam/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py", line 955, in Get
          download=download)
        File ".../tmp/virtualenvs/py3-env/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 722, in _RunMethod
          return self.ProcessHttpResponse(method_config, http_response, request)
        File ".../tmp/virtualenvs/py3-env/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 728, in ProcessHttpResponse
          self.__ProcessHttpResponse(method_config, http_response, request))
        File ".../tmp/virtualenvs/py3-env/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 611, in __ProcessHttpResponse
          response_type, http_response.content)
        File ".../tmp/virtualenvs/py3-env/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 442, in DeserializeMessage
          message = encoding.JsonToMessage(response_type, data)
        File ".../tmp/virtualenvs/py3-env/lib/python3.5/site-packages/apitools/base/py/encoding.py", line 104, in JsonToMessage
          return _ProtoJsonApiTools.Get().decode_message(message_type, message)
        File ".../tmp/virtualenvs/py3-env/lib/python3.5/site-packages/apitools/base/py/encoding.py", line 290, in decode_message
          message_type, result)
        File ".../tmp/virtualenvs/py3-env/lib/python3.5/site-packages/apitools/base/protorpclite/protojson.py", line 210, in decode_message
          dictionary = json.loads(encoded_message)
        File "/usr/lib/python3.5/json/__init__.py", line 312, in loads
          s.__class__.__name__))
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                markflyhigh Mark Liu
                Reporter:
                markflyhigh Mark Liu
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 4h 50m
                  4h 50m