Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24417 Build and Run Spark on JDK11
  3. SPARK-26134

Upgrading Hadoop to 2.7.4 to fix java.version problem

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • Spark Core
    • None

    Description

      When I ran spark-shell on JDK11+28(2018-09-25), It failed with the error below.

      Exception in thread "main" java.lang.ExceptionInInitializerError
      	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
      	at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
      	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
      	at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
      	at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:791)
      	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:761)
      	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:634)
      	at org.apache.spark.util.Utils$.$anonfun$getCurrentUserName$1(Utils.scala:2427)
      	at scala.Option.getOrElse(Option.scala:121)
      	at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2427)
      	at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:79)
      	at org.apache.spark.deploy.SparkSubmit.secMgr$lzycompute$1(SparkSubmit.scala:359)
      	at org.apache.spark.deploy.SparkSubmit.secMgr$1(SparkSubmit.scala:359)
      	at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$9(SparkSubmit.scala:367)
      	at scala.Option.map(Option.scala:146)
      	at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:367)
      	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:143)
      	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
      	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:927)
      	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:936)
      	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
      Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 2
      	at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3319)
      	at java.base/java.lang.String.substring(String.java:1874)
      	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:52)
      

      This is a Hadoop issue that fails to parse some java.version. It has been fixed from Hadoop-2.7.4(see HADOOP-14586).

      Note, Hadoop-2.7.5 or upper have another problem with Spark (SPARK-25330). So upgrading to 2.7.4 would be fine for now.

      Attachments

        Activity

          People

            tasanuma Takanobu Asanuma
            tasanuma Takanobu Asanuma
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: