Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-7950

HiveThriftServer2.startWithContext() doesn't set "spark.sql.hive.version"

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 1.2.0, 1.2.1, 1.2.2, 1.3.0, 1.3.1, 1.4.0
    • 1.4.0
    • SQL
    • None
    • Simba Spark SQL ODBC driver 1.0.8.1006

    Description

      While testing the newly released Simba Spark SQL ODBC driver 1.0.8.1006 against 1.4.0-SNAPSHOT, we found that if HiveThriftServer2 is started with HiveThriftServer2.startWithContext(), then simple queries like

      SELECT * FROM src
      

      fail with the following error message (need to turn on ODBC trace log):

      DIAG [S0002] [Simba][SQLEngine] (31740) Table or view not found: SPARK..src
      

      However, JDBC client like Beeline is fine. Also, if the server is started via sbin/start-thriftserver.sh, both ODBC and JDBC work fine.

      The reason for this failure is that, HiveThriftServer2.startWithContext() doesn't properly set the "spark.sql.hive.version" property. It seems that Simba ODBC driver 1.0.8.1006 behaves differently when this property is missing. What I observed is that, in this case, the ODBC driver issues a GetColumns command, which isn't overriden in Spark HiveThriftServer2, and this falls back to original Hive code path, which results in unexpected behavior.

      Attachments

        Activity

          People

            lian cheng Cheng Lian
            lian cheng Cheng Lian
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: