Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-6984 Python 3.7 Support
  3. BEAM-8397

DataflowRunnerTest.test_remote_runner_display_data sometimes fails due to infinite recursion during pickling on Python 3.7.

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.17.0
    • Component/s: sdk-py-core
    • Labels:
      None

      Description

      `python ./setup.py test -s apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest.test_remote_runner_display_data` passes.
      `tox -e py37-gcp` passes if Beam depends on dill==0.3.0, but fails if Beam depends on dill==0.3.1.1.`python ./setup.py nosetests --tests 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data` fails currently if run on master.

      The failure indicates infinite recursion during pickling:

      test_remote_runner_display_data (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest) ... Fatal Python error: Cannot recover from stack overflow.
      
      Current thread 0x00007f9d700ed740 (most recent call first):
        File "/usr/lib/python3.7/pickle.py", line 479 in get
        File "/usr/lib/python3.7/pickle.py", line 497 in save
        File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
        File "/usr/lib/python3.7/pickle.py", line 504 in save
        File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
        File "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py", line 1394 in save_function
        File "/usr/lib/python3.7/pickle.py", line 504 in save
        File "/usr/lib/python3.7/pickle.py", line 882 in _batch_setitems
        File "/usr/lib/python3.7/pickle.py", line 856 in save_dict
        File "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py", line 910 in save_module_dict
        File "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py", line 198 in new_save_module_dict
        File "/usr/lib/python3.7/pickle.py", line 504 in save
        File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
        File "/usr/lib/python3.7/pickle.py", line 504 in save
        File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
        File "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py", line 114 in wrapper
        File "/usr/lib/python3.7/pickle.py", line 504 in save
        File "/usr/lib/python3.7/pickle.py", line 771 in save_tuple
        File "/usr/lib/python3.7/pickle.py", line 504 in save
        File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
        File "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py", line 1137 in save_cell
        File "/usr/lib/python3.7/pickle.py", line 504 in save
        File "/usr/lib/python3.7/pickle.py", line 771 in save_tuple
        File "/usr/lib/python3.7/pickle.py", line 504 in save
        File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
        File "/usr/lib/python3.7/pickle.py", line 504 in save
        File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
        File "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py", line 1394 in save_function
        File "/usr/lib/python3.7/pickle.py", line 504 in save
        File "/usr/lib/python3.7/pickle.py", line 882 in _batch_setitems
        File "/usr/lib/python3.7/pickle.py", line 856 in save_dict
        File "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py", line 910 in save_module_dict
        File "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py", line 198 in new_save_module_dict
      ...
      

      cc: yoshiki obata

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tvalentyn Valentyn Tymofieiev
                Reporter:
                tvalentyn Valentyn Tymofieiev
              • Votes:
                1 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2.5h
                  2.5h