Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25731

Spark Structured Streaming Support for Kafka 2.0

    XMLWordPrintableJSON

Details

    Description

      https://github.com/apache/spark/tree/master/external

      As far as I can see,
      This doesn't have support for newly release kafka2.0,
      support is available only till kafka-0-10.

      If we use the 

      "org.apache.spark" %% "spark-streaming-kafka-0-10" % "2.3.0"

      for kafka2.0, below is the error I get

      11:46:18.061 [stream execution thread for [id = e393ea37-8009-4ce0-b996-94f767994fb8, runId = bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-2, groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0] Completed connection to node -1. Fetching API versions.
      11:46:18.061 [stream execution thread for [id = e393ea37-8009-4ce0-b996-94f767994fb8, runId = bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-2, groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0] Initiating API versions fetch from node -1.
      11:46:18.452 [stream execution thread for [id = e393ea37-8009-4ce0-b996-94f767994fb8, runId = bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG org.apache.kafka.common.network.Selector - [Consumer clientId=consumer-2, groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0] Connection with kafka-muhammad-45e0.aivencloud.com/18.203.67.147 disconnected
      java.io.EOFException: null
      at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:119)
      at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:335)
      at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:296)

       

       

      I might be wrong, but this is the best option I thought to open an issue. 

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              meetchandan Chandan
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: