Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Duplicate
-
3.0.0
-
None
-
None
Description
Now Kafka source provides options to specify custom offset per topic partition to set where to start reading from, and where to stop reading.
I'd like to introduce new options to specify timestamp per topic (not topic partition - we can support but it would be unlikely for us to set timestamp per partition) to fetch offset via timestamp and start reading from and stop reading.
The characteristic of new options would be very similar to existing options. For example, in streaming query, ending timestamp option would not be valid, and starting timestamp option only affects when query starts - if query restores from checkpoint the option would not be in effect.
The new timestamp option would take precedence over offset option.
Attachments
Issue Links
- duplicates
-
SPARK-26848 Introduce new option to Kafka source - specify timestamp to start and end offset
- Resolved