Details
-
Improvement
-
Status: Open
-
P3
-
Resolution: Unresolved
-
None
-
None
-
None
Description
Currently, Beam model doesn't have any information on the order of input/output PCollections from PTransforms. Therefore, PCollectionList needs to be converted to PCollectionTuple when it goes across the cross-language boundaries (or even in the same language, whenever it is converted between in-memory object and proto) and it's impossible to recreate PCollectionList from proto with the original order. The possible workaround is just to use PCollectionTuple with integer id (starting from 0 like indexes) instead of PCollectionList. In that case, we should first well-define how we generate proto from PCollectionList since each SDK uses a different convention.