Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-10310 Kafka Raft Snapshot
  3. KAFKA-12155

Delay increasing the log start offset

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • None
    • replication
    • None

    Description

      The implementation in https://github.com/apache/kafka/pull/9816 increases the log start offset as soon as a snapshot is created that is greater than the log start offset. This is correct but causes some inefficiency in some cases.

      1. Any follower, voters or observers, with an end offset between the leader's log start offset and the leader's latest snapshot will get invalidated. This will cause those follower to fetch the new snapshot and reload it's state machine.
      2. Any Listener or state machine that has a nextExpectedOffset() less than the latest snapshot will get invalidated. This will cause the state machine to have to reload its state from the latest snapshot.

      To minimize the frequency of these reloads KIP-630 proposes adding the following configuration:

      • metadata.start.offset.lag.time.max.ms - The maximum amount of time that leader will wait for an offset to get replicated to all of the live replicas before advancing the LogStartOffset. See section “When to Increase the LogStartOffset”. The default is 7 days.

      This description and implementation should be extended to also apply to the state machine, or Listener. The local log start offset should be increased when all of the ListenerContext's nextExpectedOffset() is greater than the offset of the latest snapshot.

      Attachments

        Issue Links

          Activity

            People

              mumrah David Arthur
              jagsancio Jose Armando Garcia Sancio
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: