Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-47013

Document the config spark.sql.streaming.minBatchesToRetain

    XMLWordPrintableJSON

Details

    • Documentation
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 3.5.0
    • None
    • Structured Streaming
    • None

    Description

      Add the config spark.sql.streaming.minBatchesToRetain to the streaming docs page which basically controls the minimum number of batches that must be retained and made recoverable.

      This would also help control the lifecycle of the state files held in the checkpoint folder i.e, State files are cleaned up based on the config spark.sql.streaming.minBatchesToRetain

      Attachments

        Activity

          People

            Unassigned Unassigned
            lingesh Lingeshwaran Radhakrishnan
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: