Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28258

Incopatibility betweek spark docker image and hadoop 3.2 and azure tools

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Not A Problem
    • 2.4.3
    • None
    • Kubernetes, Spark Core
    • None

    Description

      Currently the docker images generated by the distro uses openjdk8 based on alpine.
      This means that the version shipped of libssl is 1.1.1b-r1:
       

      sh-4.4# apk list | grep ssl
      libssl1.1-1.1.1b-r1 x86_64 {openssl} (OpenSSL) [installed] 
      

      The hadoop distro ships wildfly-openssl-1.0.4.Final.jar, which is affected by https://issues.jboss.org/browse/JBEAP-16425.

      This results on error running on the executor:

      2019-07-04 22:32:40,339 INFO openssl.SSL: WFOPENSSL0002 OpenSSL Version OpenSSL 1.1.1b 26 Feb 2019
      2019-07-04 22:32:40,363 WARN streaming.FileStreamSink: Error while looking for metadata directory.
      Exception in thread "main" java.lang.NullPointerException
       at org.wildfly.openssl.CipherSuiteConverter.toJava(CipherSuiteConverter.java:284)
      

      In my tests creating a Docker image with an updated version of wildfly, solves the issue: 1.0.7.Final

      Not sure if this is an Spark problem, if so, where would be the right place to solve it. 

      It seems they may take care of it in Hadoop directly, but tickets are open.

      https://issues.apache.org/jira/browse/HADOOP-16410

      https://issues.apache.org/jira/browse/HADOOP-16405

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              jlpedrosa Jose Luis Pedrosa
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: