Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-16364

MM2 High-Resolution Offset Translation

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • None
    • None
    • mirrormaker

    Description

      The current OffsetSyncStore implementation https://github.com/apache/kafka/blob/8b72a2c72f09838fdd2e7416c98d30fe876b4078/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/OffsetSyncStore.java#L57 stores a sparse index of offset syncs. This attempts to strike a balanced default behavior between offset translation availability, memory usage, and throughput on the offset syncs topic.

      However, this balanced default behavior is not good enough in all circumstances. When precise offset translation is needed away from the end of the topic, such as for consumer groups with persistent lag, offset translation can be more precise. Users should have a way to configure high-precision offset translation, either through additional memory usage or re-reading the offset syncs topic.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              gharris1727 Greg Harris
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: