Details
-
Bug
-
Status: Open
-
P3
-
Resolution: Unresolved
-
None
-
None
-
None
Description
The elements in TFX intermediate collections are dictionaries of (typically single-element) numpy arrays, which are (relatively) expensive to serialize (e.g. using pickle for the numpy wrapper of a primitive int/float, repeating the column names in every element).
Though it'd be good to use a better intermediate representation, this is exacerbated because the fusion algorithm does not pack as much possible into executable stages.