Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-1251 Python 3 Support
  3. BEAM-6532

BigQuery IO does not work in Python 3

Details

    • Sub-task
    • Status: Resolved
    • P0
    • Resolution: Fixed
    • None
    • 2.11.0
    • sdk-py-core
    • None

    Description

      Suspecting an issue in apitools library, similar to one markflyhigh discovered for GCS in BEAM-6154.

      WARNING:root:Retry with exponential backoff: waiting for 7.8972419555895 seconds before retrying get_table_location because we caught exception: TypeError: the JSON object must be str, not 'b
      ytes'                                                                                                                                                                                          
       Traceback for above exception (most recent call last):                                                                                                                                        
        File "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/utils/retry.py", line 184, in wrapper                                                                                   
          return fun(*args, **kwargs)                                                                                                                                                                
        File "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/io/gcp/bigquery_tools.py", line 421, in get_table_location                                                              
          table = self._get_table(project_id, dataset_id, table_id)                                                                                                                                  
        File "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/utils/retry.py", line 197, in wrapper                                                                                   
          raise_with_traceback(exn, exn_traceback)                                                                                                                                                   
        File "/home/valentyn/tmp/venv/py3env/lib/python3.5/site-packages/future/utils/__init__.py", line 421, in raise_with_traceback                                                                
          raise exc.with_traceback(traceback)                                                                                                                                                        
        File "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/utils/retry.py", line 184, in wrapper                                                                                   
          return fun(*args, **kwargs)                                                                                                                                                                
        File "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/io/gcp/bigquery_tools.py", line 333, in _get_table                                                                      
          response = self.client.tables.Get(request)                                                                                                                                                 
        File "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py", line 579, in Get                                               
          config, request, global_params=global_params)                                                                                                                                              
        File "/home/valentyn/tmp/venv/py3env/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 731, in _RunMethod                                                                      
          return self.ProcessHttpResponse(method_config, http_response, request)                                                                                                                     
        File "/home/valentyn/tmp/venv/py3env/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 737, in ProcessHttpResponse                                                             
          self.__ProcessHttpResponse(method_config, http_response, request))                                                                                                                         
        File "/home/valentyn/tmp/venv/py3env/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 620, in __ProcessHttpResponse                                                           
          return self.__client.DeserializeMessage(response_type, content)                                                                                                                            
        File "/home/valentyn/tmp/venv/py3env/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 446, in DeserializeMessage                                                              
          message = encoding.JsonToMessage(response_type, data)                                                                                                                                      
        File "/home/valentyn/tmp/venv/py3env/lib/python3.5/site-packages/apitools/base/py/encoding_helper.py", line 123, in JsonToMessage                                                            
          return _ProtoJsonApiTools.Get().decode_message(message_type, message)                                                                                                                      
        File "/home/valentyn/tmp/venv/py3env/lib/python3.5/site-packages/apitools/base/py/encoding_helper.py", line 309, in decode_message                                                           
          message_type, result)                                                                                                                                                                      
        File "/home/valentyn/tmp/venv/py3env/lib/python3.5/site-packages/apitools/base/protorpclite/protojson.py", line 212, in decode_message                                                       
          dictionary = json.loads(encoded_message)                                                                                                                                                   
        File "/usr/lib/python3.5/json/__init__.py", line 312, in loads                                                                                                                               
          s.__class__.__name__))                                                                                                                                                                     
      
      

      Attachments

        Issue Links

          Activity

            People

              tvalentyn Valentyn Tymofieiev
              tvalentyn Valentyn Tymofieiev
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h
                  1h