Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31614

Unable to write data into hive table using Spark via Hive JDBC driver Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Incomplete
    • 2.3.1
    • None
    • Spark Shell, Spark Submit
    • HDP3.0,spark 2.3.1,hadoop 3.1.1

    • Patch, Important

    Description

      I'm trying to wrire data into hive table using a JDBC connection to Hive. Unfortunately, when I write data that resides in every column I get the following error:

      org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: ParseException line 1:36 cannot recognize input near '.' 'aname' 'TEXT' in column type
      at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:255)
      at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:241)

      1)  On Hive create a simple table,its name is "test",it have three column(aname,score,banji),their type both are "String"

      2)important code:

      object HiveDialect extends JdbcDialect

      { override def canHandle(url: String): Boolean = url.startsWith("jdbc:hive2")|| url.contains("hive2") override def quoteIdentifier(colName: String): String = s"$colName" }

      -------------------------------------------------------------------

      object callOffRun {
      def main(args: Array[String]): Unit =

      { val spark = SparkSession.builder().enableHiveSupport().getOrCreate() JdbcDialects.registerDialect(HiveDialect) val props = new Properties() props.put("driver","org.apache.hive.jdbc.HiveDriver") props.put("user","username") props.put("password","password") props.put("fetchsize","20") val table=spark.read.jdbc("jdbc:hive2://xxxxxxxx:10000","test",props) table.write.jdbc("jdbc:hive2://xxxxxxxx:10000", "resulttable", props) }

      }

      3)spark-submit ,After running,When table write,it have error 

      4)table.count() have result 

      5) i try some method to write data into table,They all reported the same error

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            liuzhang liuzhang
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - 1,560h
                1,560h
                Remaining:
                Remaining Estimate - 1,560h
                1,560h
                Logged:
                Time Spent - Not Specified
                Not Specified