Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-7897

Invalid use of epoch cache with old message format versions

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 2.0.1, 2.1.0
    • 1.1.2, 2.2.0, 2.1.1, 2.0.2
    • None
    • None

    Description

      Message format downgrades are not supported, but they generally work as long as broker/clients at least can continue to parse both message formats. After a downgrade, the truncation logic should revert to using the high watermark, but currently we use the existence of any cached epoch as the sole prerequisite in order to leverage OffsetsForLeaderEpoch. This has the effect of causing a massive truncation after startup which causes re-replication.

      I think our options to fix this are to either 1) clear the cache when we notice a downgrade, or 2) forbid downgrades and raise an error.

      Attachments

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. KAFKA-8012 NullPointerException while truncating at high watermark can crash replica fetcher thread

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. KAFKA-7886 Some partitions are fully truncated during recovery when log.message.format = 0.10.2 & inter.broker.protocol >= 0.11

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. KAFKA-8036 Log dir reassignment on followers fails with FileNotFoundException for the leader epoch cache on leader election

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #6232

          Web Link GitHub Pull Request #6253

          Activity

            People

              hachikuji Jason Gustafson
              hachikuji Jason Gustafson
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: