Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.0.0
-
None
Description
SPARK-20568 added options to file streaming source to clean up processed files. However, when applying these options to a directory that was written by a file streaming sink, it will make the directory not queryable any more because we delete files from the directory but they are still tracked by file sink logs.
I think we should block the options if the input source is a file streaming sink path (has "_spark_metadata" folder).
Attachments
Issue Links
- is caused by
-
SPARK-20568 Delete files after processing in structured streaming
- Resolved
- links to