Details
-
Bug
-
Status: Open
-
P3
-
Resolution: Unresolved
-
None
-
None
-
None
Description
As a workaround we could introduce a way to not perform size estimation when reading large globs. For example Java SDK has withHintMatchesManyFiles() option.
Additionally, seems like we are repeating the size estimation where the same PCollection read from a file-based source is applied to multiple PTransforms.
See following for more details.