Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Incomplete
-
2.4.0
-
None
Description
After PR https://github.com/apache/spark/pull/21048, the CheckpointFileManager interface was created to handle all structured streaming checkpointing operations and helps users to choose how they wish to write checkpointing files atomically.
StreamExecution and StreamingQueryManager still uses some FileSystem operations without using the CheckpointFileManager.
For instance,
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala#L137
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala#L392
Instead, StreamExecution and StreamingQueryManager should use CheckpointFileManager for these operations.
Attachments
Issue Links
- links to