Details
-
Bug
-
Status: Resolved
-
P2
-
Resolution: Fixed
-
2.8.0
-
None
Description
We did some profiling of a spark job and 90% of the application time was spent on side input deserialization.
For spark, an easy fix is to cache materialized side inputs per bundle. This improved running time of the profiled job from 3 hours to 30 minutes.