[BEAM-10260] Nested iterable in state cache exceeds Python recursion limit - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: P2
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 2.23.0
Component/s: sdk-py-harness
Labels:
None

Description

With state caching enabled, the changes in ~~BEAM-8298~~ lead to an unbounded nesting of state iterables when appending to a bag state, quickly resulting in an exception:

events = list(event_bag.read())  
File "/code/venvs/venv/lib/python3.6/site-packages/apache_beam/runners/worker/bundle_processor.py", line 488, in __iter__    for elem in self.first:  
File "/code/venvs/venv/lib/python3.6/site-packages/apache_beam/runners/worker/statecache.py", line 186, in __iter__    for item in value:  
File "/code/venvs/venv/lib/python3.6/site-packages/apache_beam/runners/worker/statecache.py", line 186, in __iter__    for item in value:  
File "/code/venvs/venv/lib/python3.6/site-packages/apache_beam/runners/worker/statecache.py", line 186, in __iter__    for item in value:  [Previous line repeated 977 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object

Attachments

Issue Links

is caused by

BEAM-8298 Implement state caching for side inputs for Python

Resolved

links to

GitHub Pull Request #12012

Activity

People

Assignee:: Maximilian Michels

Reporter:: Maximilian Michels

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Dates

Created:: 15/Jun/20 12:51

Updated:: 16/Jun/20 16:25

Resolved:: 16/Jun/20 14:27

Time Tracking

Estimated:

Not Specified

Remaining:

Logged:

1.5h