Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-36520

KinesisInitialPosition interface should be public

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 3.1.2
    • None
    • DStreams

    Description

      In `org.apache.spark.streaming.kinesis` there's a Java interface `KinesisInitialPosition` - for some reason it's not public, so can't be used.

      The reason it needs to be used is simple: it is the parent of `KinesisInitialPositions.Latest` and `KinesisInitialPositions.TrimHorizon`. Without using it, it's impossible to pass instances of `Latest` and `TrimHorizon` around without calling them by name.

      The workaround is to use the enum `InitialPositionInStream` instead, and a factory method `fromKinesisInitialPosition(..)` - but it doesn't seem like the correct pattern, because it belongs to Amazon Kinesis Client library and not Spark (it's currently the only import in my class which is outside of `org.apache.spark`).

      The change is trivial, so it's mainly a matter of why not to do it.

      KinesisInitialPositions: https://github.com/apache/spark/blob/master/external/kinesis-asl/src/main/java/org/apache/spark/streaming/kinesis/KinesisInitialPositions.java
      KinesisInitialPosition: https://github.com/awslabs/amazon-kinesis-client/blob/master/amazon-kinesis-client/src/main/java/software/amazon/kinesis/common/InitialPositionInStream.java

      Attachments

        Activity

          People

            Unassigned Unassigned
            noamtm Noam Tamim
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - 1h
                1h
                Remaining:
                Remaining Estimate - 1h
                1h
                Logged:
                Time Spent - Not Specified
                Not Specified