Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-20693

Kafka+SSL: path for security related files needs to be different for driver and executors

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.1.1
    • Fix Version/s: None
    • Component/s: Input/Output
    • Labels:

      Description

      When consuming/producing from Kafka with security enable (SSL), you need to refer to security related files (keystore and truststore) in the configuration of the KafkaDirectStream.

      If the scenario is YARN-client mode, you would need to distribute these files, it can be achieved with --files argument. Now, what is the path to these files? taking into account that driver and executors interact with Kafka.

      When these files are accessed from the driver, you need to provide the local path to them. When they are accessed from the executors, you need to provide the name of the file that has been distributed with --files.

      The problem is that you can only configure one value for the path to these files.

      Proposed configurations here: http://www.opencore.com/blog/2017/1/spark-2-0-streaming-from-ssl-kafka-with-hdp-2-4/
      works because both paths are the same (./truststore.jks). But if different, I do not think there is a way to configure Kafka+SSL

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              DLanza Daniel Lanza GarcĂ­a
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: