Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-35804

can't read external hive table on spark

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Incomplete
    • 2.3.2
    • None
    • PySpark, Spark Core, Spark Shell
    • hdp 3.1.4 

      hive-hcatalog-core-3.1.0.3.1.4.0-315.jar & hive-hcatalog-core-3.1.2 both I've tried

       

    Description

      I create a external hive table with hdfs file which is formatted as json string.

      I can read the data field of this hive table with the help of  org.apache.hive.hcatalog.data.JsonSerDe which is packed in hive-hcatalog-core.jar in hive shell.

      But when I try to use the spark (pyspark ,spark-shell or whatever) ,I just can't read it.

      It gave me a error  Table: Unable to get field from serde: org.apache.hive.hcatalog.data.JsonSerDe

      I've copied the jar (hive-hcatalog-core.jar) to $spark_home/jars and yarn libs and rerun ,there is no effect,even use --jars $jar_path/hive-hcatalog-core.jar.But  when I browse the webpage of spark task ,I can actually find the jar in the env list.

       

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            mochapuff cao zhiyu
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: