Details
-
New Feature
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.1.0, 2.2.1
-
None
Description
It would be great to be able to delete files after processing them with structured streaming.
For example, I am reading in a bunch of JSON files and converting them into Parquet. If the JSON files are not deleted after they are processed, it quickly fills up my hard drive. I originally posted this on Stack Overflow and was recommended to make a feature request for it.
Attachments
Issue Links
- causes
-
SPARK-29953 File stream source cleanup options may break a file sink output
- Resolved
- links to