Details
-
New Feature
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
None
-
None
Description
The current OffsetSyncStore implementation https://github.com/apache/kafka/blob/8b72a2c72f09838fdd2e7416c98d30fe876b4078/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/OffsetSyncStore.java#L57 stores a sparse index of offset syncs. This attempts to strike a balanced default behavior between offset translation availability, memory usage, and throughput on the offset syncs topic.
However, this balanced default behavior is not good enough in all circumstances. When precise offset translation is needed away from the end of the topic, such as for consumer groups with persistent lag, offset translation can be more precise. Users should have a way to configure high-precision offset translation, either through additional memory usage or re-reading the offset syncs topic.
Attachments
Issue Links
- is related to
-
KAFKA-16291 Mirrormaker2 wrong checkpoints
- Open
-
KAFKA-15905 Restarts of MirrorCheckpointTask should not permanently interrupt offset translation
- Resolved
- relates to
-
KAFKA-16641 MM2 offset translation should interpolate between sparse OffsetSyncs
- Open