Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26848

Introduce new option to Kafka source - specify timestamp to start and end offset

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • SQL
    • None

    Description

      Kafka source (for SQL/SS) provides options to set specific offset per topic partition to let source starts reading from start offsets and ends reading until end offsets. ("startingOffsets" and "endingOffsets" in below document.)

      http://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html

      I'd like to introduce new options "startingOffsetsByTimestamp" and "endingOffsetsByTimestamp" to set specific timestamp per topic (since we're unlikely to set the different value per partition) to let source starts reading from offsets which have equal of greater timestamp, and ends reading until offsets which have equal of greater timestamp.

      The option would be optional of course, and have a preference to apply the options 1) timestamp option applies first, and if it doesn't exist, offset option applies.

      Attachments

        Issue Links

          Activity

            People

              kabhwan Jungtaek Lim
              kabhwan Jungtaek Lim
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: