Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Incomplete
-
2.1.1
-
None
Description
When consuming/producing from Kafka with security enable (SSL), you need to refer to security related files (keystore and truststore) in the configuration of the KafkaDirectStream.
If the scenario is YARN-client mode, you would need to distribute these files, it can be achieved with --files argument. Now, what is the path to these files? taking into account that driver and executors interact with Kafka.
When these files are accessed from the driver, you need to provide the local path to them. When they are accessed from the executors, you need to provide the name of the file that has been distributed with --files.
The problem is that you can only configure one value for the path to these files.
Proposed configurations here: http://www.opencore.com/blog/2017/1/spark-2-0-streaming-from-ssl-kafka-with-hdp-2-4/
works because both paths are the same (./truststore.jks). But if different, I do not think there is a way to configure Kafka+SSL