Details
-
Documentation
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
3.5.0
-
None
-
None
Description
Add the config spark.sql.streaming.minBatchesToRetain to the streaming docs page which basically controls the minimum number of batches that must be retained and made recoverable.
This would also help control the lifecycle of the state files held in the checkpoint folder i.e, State files are cleaned up based on the config spark.sql.streaming.minBatchesToRetain