Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Fixed
-
1.7.2, 1.8.1, 1.9.1
Description
Currently, BatchTask and DataSourceTask only allow outputting elements in close method of "rich" operators, that they directly execute.
Task workflow is as follows:
1) open "head" driver (calls "open" method on udf)
2) open chained drivers
3) run "head" driver
4) close "head" driver (calls "close" method on udf)
5) close output collector (no elements can be collected after this point)
6) close chained drivers
In order to properly support outputs from close method, we want to switch 6) and 5). We also need to tweak implementation of Reduce / Combine chained drivers, because they dispose sorters in closeTask method (this should be done in the close method).
This would bring huge performance improvement for Beam users, because we could properly implement bundling on batch (whole partition = single bundle).
Attachments
Issue Links
- is depended upon by
-
BEAM-8608 Chain DoFns in Flink batch runner when possible.
- Resolved
- links to