Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-1909

spark session with hive support can't be created correctly

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 0.7.0
    • 0.7.0
    • Interpreters
    • None

    Description

          if (useHiveContext()) {
            if (hiveClassesArePresent()) {
              Utils.invokeMethod(builder, "enableHiveSupport");
              sparkSession = Utils.invokeMethod(builder, "getOrCreate");
              logger.info("Created Spark session with Hive support");
            } else {
              Utils.invokeMethod(builder, "config",
                  new Class[]{ String.class, String.class},
                  new Object[]{ "spark.sql.catalogImplementation", "in-memory"});
              sparkSession = Utils.invokeMethod(builder, "getOrCreate");
              logger.info("Created Spark session with Hive support");
            }
          } 
      
       /**
         * See org.apache.spark.sql.SparkSession.hiveClassesArePresent
         * @return
         */
        private boolean hiveClassesArePresent() {
          try {
            this.getClass().forName("org.apache.spark.sql.hive.HiveSessionState");
            this.getClass().forName("org.apache.spark.sql.hive.HiveSharedState");
            this.getClass().forName("org.apache.hadoop.hive.conf.HiveConf");
            return true;
          } catch (ClassNotFoundException | NoClassDefFoundError e) {
            return false;
          }
        }
      

      since "org.apache.spark.sql.hive.HiveSharedState" has been removed from spark2.1.x

      So hiveClassesArePresent will always return false
      and spark session with hive support will always be created with
      "spark.sql.catalogImplementation=in-memory"

      Maybe just remove

            this.getClass().forName("org.apache.spark.sql.hive.HiveSharedState");
      

      is OK?

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              licl lichenglin
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: