Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-20693

Kafka+SSL: path for security related files needs to be different for driver and executors

    XMLWordPrintableJSON

Details

    Description

      When consuming/producing from Kafka with security enable (SSL), you need to refer to security related files (keystore and truststore) in the configuration of the KafkaDirectStream.

      If the scenario is YARN-client mode, you would need to distribute these files, it can be achieved with --files argument. Now, what is the path to these files? taking into account that driver and executors interact with Kafka.

      When these files are accessed from the driver, you need to provide the local path to them. When they are accessed from the executors, you need to provide the name of the file that has been distributed with --files.

      The problem is that you can only configure one value for the path to these files.

      Proposed configurations here: http://www.opencore.com/blog/2017/1/spark-2-0-streaming-from-ssl-kafka-with-hdp-2-4/
      works because both paths are the same (./truststore.jks). But if different, I do not think there is a way to configure Kafka+SSL

      Attachments

        Activity

          People

            Unassigned Unassigned
            DLanza Daniel Lanza GarcĂ­a
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: