[BEAM-12071] DataFrame IO sinks do not correctly partition by window - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Triage Needed
Priority: P1
Resolution: Fixed
Affects Version/s: 2.26.0, 2.27.0, 2.28.0
Fix Version/s: 2.29.0
Component/s: dsl-dataframe, sdk-py-core
Labels:
- dataframe-api

Description

I just discovered that when processing windowed data with DataFrameTransform, only one partition is written to every window, and a single window gets every other partition/window combination.

Attachments

Issue Links

is caused by

BEAM-12072 WriteToFiles with a concrete sink can shunt data to a single window

Open

links to

GitHub Pull Request #14374

GitHub Pull Request #14383

Activity

People

Assignee:: Brian Hulette

Reporter:: Brian Hulette

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Dates

Created:: 30/Mar/21 15:53

Updated:: 13/Apr/23 11:03

Resolved:: 01/Apr/21 00:52

Time Tracking

Estimated:

Not Specified

Remaining:

Logged:

3.5h