Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26421

Spark2.4.0 integration hadoop3.1.1 causes hive sql not to use，just in idea local mode

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Duplicate
    • 2.4.0
    • None
    • Deploy
    • None

    • ).idea maven project

      ).jdk 1.8.0_191

      ).hadoop 3.1.1

      ).spark 2.4.0

    • Hide
      <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
          <modelVersion>4.0.0</modelVersion>
          <groupId>com.opensource</groupId>
          <artifactId>spark-scala-maven-2.4.0</artifactId>
          <version>1.0-SNAPSHOT</version>
          <inceptionYear>2008</inceptionYear>

          <properties>
              <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
              <scala.version>2.11.12</scala.version>
              <java.version>1.8.191</java.version>
              <spark.version>2.4.0</spark.version>
              <hadoop.version>3.1.1</hadoop.version>
              <!--<hadoop.version>2.9.2</hadoop.version>-->
              <spark.scala.version>2.11</spark.scala.version>


              <PermGen>64m</PermGen>
              <MaxPermGen>512m</MaxPermGen>
              <CodeCacheSize>512m</CodeCacheSize>

          </properties>



          <pluginRepositories>
              <pluginRepository>
                  <id>scala-tools.org</id>
                  <name>Scala-Tools Maven2 Repository</name>
                  <url>http://scala-tools.org/repo-releases&lt;/url>
              </pluginRepository>
          </pluginRepositories>

          <dependencies>
              <dependency>
                  <groupId>org.scala-lang</groupId>
                  <artifactId>scala-library</artifactId>
                  <version>${scala.version}</version>
              </dependency>

              <dependency>
                  <groupId>org.scala-lang</groupId>
                  <artifactId>scala-reflect</artifactId>
                  <version>${scala.version}</version>
              </dependency>
              <dependency>
                  <groupId>org.scala-lang</groupId>
                  <artifactId>scala-compiler</artifactId>
                  <version>${scala.version}</version>
              </dependency>



              <!-- spark core-->
              <dependency>
                  <groupId>org.apache.spark</groupId>
                  <artifactId>spark-core_${spark.scala.version}</artifactId>
                  <version>${spark.version}</version>
              </dependency>


              <!-- org.apache.spark/spark-sql -->
              <dependency>
                  <groupId>org.apache.spark</groupId>
                  <artifactId>spark-sql_${spark.scala.version}</artifactId>
                  <version>${spark.version}</version>
              </dependency>

              <dependency>
                  <groupId>org.apache.spark</groupId>
                  <artifactId>spark-hive_${spark.scala.version}</artifactId>
                  <version>${spark.version}</version>
              </dependency>

         <!-- <dependency>
                  <groupId>org.apache.spark</groupId>
                  <artifactId>spark-hive-thriftserver_2.11</artifactId>
                  <version>2.4.0</version>

              </dependency>
             -->


              <!-- hadoop-client-->
              <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client -->
              <dependency>
                  <groupId>org.apache.hadoop</groupId>
                  <artifactId>hadoop-client</artifactId>
                  <version>${hadoop.version}</version>
              </dependency>

              <dependency>
                  <groupId>org.apache.hadoop</groupId>
                  <artifactId>hadoop-common</artifactId>
                  <version>${hadoop.version}</version>
              </dependency>

              <dependency>
                  <groupId>org.apache.hadoop</groupId>
                  <artifactId>hadoop-hdfs</artifactId>
                  <version>${hadoop.version}</version>
              </dependency>

              <dependency>
                  <groupId>org.apache.hadoop</groupId>
                  <artifactId>hadoop-yarn-client</artifactId>
                  <version>${hadoop.version}</version>
              </dependency>


              <dependency>
                  <groupId>junit</groupId>
                  <artifactId>junit</artifactId>
                  <version>4.4</version>
                  <scope>test</scope>
              </dependency>

              <dependency>
                  <groupId>org.specs</groupId>
                  <artifactId>specs</artifactId>
                  <version>1.2.5</version>
                  <scope>test</scope>
              </dependency>




              <dependency>
                  <groupId>mysql</groupId>
                  <artifactId>mysql-connector-java</artifactId>
                  <version>8.0.13</version>
              </dependency>



          </dependencies>

          <build>
              <sourceDirectory>src/main/scala</sourceDirectory>
              <testSourceDirectory>src/test/scala</testSourceDirectory>
              <resources>
                  <resource><directory>src/main/resource</directory></resource>
              </resources>
              <plugins>

                  <plugin>
                  <groupId>net.alchim31.maven</groupId>
                  <artifactId>scala-maven-plugin</artifactId>
                  <version>3.2.2</version>
                  <executions>
                      <execution>
                          <id>eclipse-add-source</id>
                          <goals>
                              <goal>add-source</goal>
                          </goals>
                      </execution>
                      <execution>
                          <id>scala-compile-first</id>
                          <phase>process-resources</phase>
                          <goals>
                              <goal>compile</goal>
                          </goals>
                      </execution>
                      <execution>
                          <id>scala-test-compile-first</id>
                          <phase>process-test-resources</phase>
                          <goals>
                              <goal>testCompile</goal>
                          </goals>
                      </execution>
                      <execution>
                          <id>attach-scaladocs</id>
                          <phase>verify</phase>
                          <goals>
                              <goal>doc-jar</goal>
                          </goals>
                      </execution>
                  </executions>
                  <configuration>
                      <scalaVersion>${scala.version}</scalaVersion>
                      <recompileMode>incremental</recompileMode>
                      <useZincServer>true</useZincServer>
                      <args>
                          <arg>-unchecked</arg>
                          <arg>-deprecation</arg>
                          <arg>-feature</arg>
                      </args>
                      <jvmArgs>
                          <jvmArg>-Xms1024m</jvmArg>
                          <jvmArg>-Xmx1024m</jvmArg>
                          <jvmArg>-XX:PermSize=${PermGen}</jvmArg>
                          <jvmArg>-XX:MaxPermSize=${MaxPermGen}</jvmArg>
                          <jvmArg>-XX:ReservedCodeCacheSize=${CodeCacheSize}</jvmArg>
                      </jvmArgs>
                      <javacArgs>
                          <javacArg>-source</javacArg>
                          <javacArg>${java.version}</javacArg>
                          <javacArg>-target</javacArg>
                          <javacArg>${java.version}</javacArg>
                          <javacArg>-Xlint:all,-serial,-path</javacArg>
                      </javacArgs>
                  </configuration>
              </plugin>


                  <plugin>
                      <groupId>org.apache.maven.plugins</groupId>
                      <artifactId>maven-compiler-plugin</artifactId>
                      <version>3.3</version>
                      <configuration>
                          <source>${java.version}</source>
                          <target>${java.version}</target>
                          <encoding>UTF-8</encoding>
                          <maxmem>1024m</maxmem>
                          <fork>true</fork>
                          <compilerArgs>
                              <arg>-Xlint:all,-serial,-path</arg>
                          </compilerArgs>
                      </configuration>
                  </plugin>

              </plugins>
          </build>
         <!-- <reporting>
              <plugins>
                  <plugin>
                      <groupId>org.scala-tools</groupId>
                      <artifactId>maven-scala-plugin</artifactId>
                      <version>2.15.2</version>

                  </plugin>


              </plugins>
          </reporting>-->


      </project>
      Show
      <project xmlns=" http://maven.apache.org/POM/4.0.0 " xmlns:xsi=" http://www.w3.org/2001/XMLSchema-instance " xsi:schemaLocation=" http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd ">     <modelVersion>4.0.0</modelVersion>     <groupId>com.opensource</groupId>     <artifactId>spark-scala-maven-2.4.0</artifactId>     <version>1.0-SNAPSHOT</version>     <inceptionYear>2008</inceptionYear>     <properties>         <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>         <scala.version>2.11.12</scala.version>         <java.version>1.8.191</java.version>         <spark.version>2.4.0</spark.version>         <hadoop.version>3.1.1</hadoop.version>         <!--<hadoop.version>2.9.2</hadoop.version>-->         <spark.scala.version>2.11</spark.scala.version>         <PermGen>64m</PermGen>         <MaxPermGen>512m</MaxPermGen>         <CodeCacheSize>512m</CodeCacheSize>     </properties>     <pluginRepositories>         <pluginRepository>             <id>scala-tools.org</id>             <name>Scala-Tools Maven2 Repository</name>             <url> http://scala-tools.org/repo-releases&lt;/url >         </pluginRepository>     </pluginRepositories>     <dependencies>         <dependency>             <groupId>org.scala-lang</groupId>             <artifactId>scala-library</artifactId>             <version>${scala.version}</version>         </dependency>         <dependency>             <groupId>org.scala-lang</groupId>             <artifactId>scala-reflect</artifactId>             <version>${scala.version}</version>         </dependency>         <dependency>             <groupId>org.scala-lang</groupId>             <artifactId>scala-compiler</artifactId>             <version>${scala.version}</version>         </dependency>         <!-- spark core-->         <dependency>             <groupId>org.apache.spark</groupId>             <artifactId>spark-core_${spark.scala.version}</artifactId>             <version>${spark.version}</version>         </dependency>         <!-- org.apache.spark/spark-sql -->         <dependency>             <groupId>org.apache.spark</groupId>             <artifactId>spark-sql_${spark.scala.version}</artifactId>             <version>${spark.version}</version>         </dependency>         <dependency>             <groupId>org.apache.spark</groupId>             <artifactId>spark-hive_${spark.scala.version}</artifactId>             <version>${spark.version}</version>         </dependency>    <!-- <dependency>             <groupId>org.apache.spark</groupId>             <artifactId>spark-hive-thriftserver_2.11</artifactId>             <version>2.4.0</version>         </dependency>        -->         <!-- hadoop-client-->         <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client -->         <dependency>             <groupId>org.apache.hadoop</groupId>             <artifactId>hadoop-client</artifactId>             <version>${hadoop.version}</version>         </dependency>         <dependency>             <groupId>org.apache.hadoop</groupId>             <artifactId>hadoop-common</artifactId>             <version>${hadoop.version}</version>         </dependency>         <dependency>             <groupId>org.apache.hadoop</groupId>             <artifactId>hadoop-hdfs</artifactId>             <version>${hadoop.version}</version>         </dependency>         <dependency>             <groupId>org.apache.hadoop</groupId>             <artifactId>hadoop-yarn-client</artifactId>             <version>${hadoop.version}</version>         </dependency>         <dependency>             <groupId>junit</groupId>             <artifactId>junit</artifactId>             <version>4.4</version>             <scope>test</scope>         </dependency>         <dependency>             <groupId>org.specs</groupId>             <artifactId>specs</artifactId>             <version>1.2.5</version>             <scope>test</scope>         </dependency>         <dependency>             <groupId>mysql</groupId>             <artifactId>mysql-connector-java</artifactId>             <version>8.0.13</version>         </dependency>     </dependencies>     <build>         <sourceDirectory>src/main/scala</sourceDirectory>         <testSourceDirectory>src/test/scala</testSourceDirectory>         <resources>             <resource><directory>src/main/resource</directory></resource>         </resources>         <plugins>             <plugin>             <groupId>net.alchim31.maven</groupId>             <artifactId>scala-maven-plugin</artifactId>             <version>3.2.2</version>             <executions>                 <execution>                     <id>eclipse-add-source</id>                     <goals>                         <goal>add-source</goal>                     </goals>                 </execution>                 <execution>                     <id>scala-compile-first</id>                     <phase>process-resources</phase>                     <goals>                         <goal>compile</goal>                     </goals>                 </execution>                 <execution>                     <id>scala-test-compile-first</id>                     <phase>process-test-resources</phase>                     <goals>                         <goal>testCompile</goal>                     </goals>                 </execution>                 <execution>                     <id>attach-scaladocs</id>                     <phase>verify</phase>                     <goals>                         <goal>doc-jar</goal>                     </goals>                 </execution>             </executions>             <configuration>                 <scalaVersion>${scala.version}</scalaVersion>                 <recompileMode>incremental</recompileMode>                 <useZincServer>true</useZincServer>                 <args>                     <arg>-unchecked</arg>                     <arg>-deprecation</arg>                     <arg>-feature</arg>                 </args>                 <jvmArgs>                     <jvmArg>-Xms1024m</jvmArg>                     <jvmArg>-Xmx1024m</jvmArg>                     <jvmArg>-XX:PermSize=${PermGen}</jvmArg>                     <jvmArg>-XX:MaxPermSize=${MaxPermGen}</jvmArg>                     <jvmArg>-XX:ReservedCodeCacheSize=${CodeCacheSize}</jvmArg>                 </jvmArgs>                 <javacArgs>                     <javacArg>-source</javacArg>                     <javacArg>${java.version}</javacArg>                     <javacArg>-target</javacArg>                     <javacArg>${java.version}</javacArg>                     <javacArg>-Xlint:all,-serial,-path</javacArg>                 </javacArgs>             </configuration>         </plugin>             <plugin>                 <groupId>org.apache.maven.plugins</groupId>                 <artifactId>maven-compiler-plugin</artifactId>                 <version>3.3</version>                 <configuration>                     <source>${java.version}</source>                     <target>${java.version}</target>                     <encoding>UTF-8</encoding>                     <maxmem>1024m</maxmem>                     <fork>true</fork>                     <compilerArgs>                         <arg>-Xlint:all,-serial,-path</arg>                     </compilerArgs>                 </configuration>             </plugin>         </plugins>     </build>    <!-- <reporting>         <plugins>             <plugin>                 <groupId>org.scala-tools</groupId>                 <artifactId>maven-scala-plugin</artifactId>                 <version>2.15.2</version>             </plugin>         </plugins>     </reporting>--> </project>

    Description

      ).Spark2.4.0 integration hadoop3.1.1 causes hive sql not to use，just in idea local mode

      ).idea maven project

      ).spark.sql connect hive 

      val spark = SparkSession
      .builder()
      .master("local")
      .appName("Spark Hive Example")
      .config("spark.sql.warehouse.dir", warehouseLocation)
      .enableHiveSupport()
      .getOrCreate()

      spark.sql("show databases").show()

       

      ).do this   error ,local environment not cluser

      ----------------------

      Exception in thread "main" java.lang.ExceptionInInitializerError
      at org.apache.hadoop.hive.conf.HiveConf.<clinit>(HiveConf.java:105)
      at java.lang.Class.forName0(Native Method)
      at java.lang.Class.forName(Class.java:348)
      at org.apache.spark.util.Utils$.classForName(Utils.scala:238)
      at org.apache.spark.sql.SparkSession$.hiveClassesArePresent(SparkSession.scala:1117)
      at org.apache.spark.sql.SparkSession$Builder.enableHiveSupport(SparkSession.scala:866)
      at com.opensource.bigdata.spark.sql.n_10_spark_hive.n_01_show_database.Run$.main(Run.scala:19)
      at com.opensource.bigdata.spark.sql.n_10_spark_hive.n_01_show_database.Run.main(Run.scala)
      Caused by: java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.1.1
      at org.apache.hadoop.hive.shims.ShimLoader.getMajorVersion(ShimLoader.java:174)
      at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:139)
      at org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:100)
      at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:368)
      ... 8 more

      Process finished with exit code 1

      Attachments

        Issue Links

          duplicates

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-23534 Spark run on Hadoop 3.0.0

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              thinktothings thinktothings
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: