Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-7739 Kafka Tiered Storage
  3. KAFKA-15351

Update log-start-offset after leader election for topics enabled with remote storage

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • None
    • 3.6.0
    • None

    Description

      Case-1:

      In the FETCH response, the leader-log-start-offset will be piggy-backed. But, there can be a scenario:

      1. Leader deleted the remote log segment and updates it's log-start-offset
      2. Before the replica-2 update it's log-start-offset via FETCH-request, the leadership changed to replica-2.
      3. There are no more eligible segments to delete from remote.
      4. The log-start-offset will be stale (referring to old log-start-offset but the data was already removed from remote)
      5. If the consumer starts to read from the beginning of the topic, it will fail to read.
        See this comment https://github.com/apache/kafka/pull/13561#discussion_r1293081560 for more details.

      Case-3: 

      When tiered storage is enabled on the topic, and the last-standing-replica is restarted, then the log-start-offset should be updated upto log-start-offset-checkpoint offset.

      Attachments

        Activity

          People

            ckamal Kamal Chandraprakash
            ckamal Kamal Chandraprakash
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: