Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-29223

Kafka source: offset by timestamp - allow specifying timestamp for "all partitions"

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0
    • Fix Version/s: None
    • Component/s: SQL, Structured Streaming
    • Labels:
      None

      Description

      This issue is a follow-up of SPARK-26848.

      In SPARK-26848, we decided to open possibility to let end users set individual timestamp per partition. But in many cases, specifying timestamp represents the intention that we would want to go back to specific timestamp and reprocess records, which should be applied to all topics and partitions.

      According to the format of `startingOffsetsByTimestamp`/`endingOffsetsByTimestamp`, while it's not intuitive to provide an option to set a global timestamp across topic, it's still intuitive to provide an option to set a global timestamp across partitions in a topic.

      This issue tracks the efforts to deal with this.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                kabhwan Jungtaek Lim
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: