Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Invalid
-
2.3.1
-
None
-
None
Description
Steps to reproduce:
- Read a directory(consisting of txt files) using spark context's wholetextfile method
- Perform transformation on the resultant paired rdd
- Perform an action(foreach) on each entry corresponding to each txt file
- Time lag can be seen between these actions in Spark UI.
The action itself is not taking that much time. There is time lag between start time for each action(excluding the time taken by the job itself). Kindly refer to the attachments
PS: This time lag is not seen when running the job in Spark 2.1.1