Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-16667

PostgreSQL metastore handling of CLOB types for COLUMNS_V2.TYPE_NAME and other field is incorrect

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.3.0, 3.0.0
    • None
    • None

    Description

      The CLOB JDO type introduced with HIVE-12274 does not work correctly with PostgreSQL. The value is written out-of-band and the LOB handle is written,as an INT, into the table. SELECTs return the INT value, which should had been read via the lo_get PG built-in, and then cast into string.

      Furthermore, the behavior is different between fields upgraded from earlier metastore versions (they retain their string storage) vs. values inserted after the upgrade (inserted as LOB roots).

      Teh code in MetasoreDirectSql.getPartitionsFromPartitionIds/extractSqlClob expects the underlying JDO/Datanucleus to map the column to a Clob but that does not happen, the value is a Java String containing the int which is the LOB root saved by PG.

      This manifests at runtime with errors like:

      hive> select * from srcpart;
      Failed with exception java.io.IOException:java.lang.IllegalArgumentException: Error: type expected at the position 0 of '24030:24031' but '24030' is found.
      

      the 24030:24031 should be 'string:string'.

      repro:

      CREATE TABLE srcpart (key STRING COMMENT 'default', value STRING COMMENT 'default') PARTITIONED BY (ds STRING, hr STRING) STORED AS TEXTFILE;
      LOAD DATA LOCAL INPATH "${hiveconf:test.data.dir}/kv1.txt" OVERWRITE INTO TABLE srcpart PARTITION (ds="2008-04-09", hr="11");
      select * from srcpart;
      

      I did not see the issue being hit by non-partitioned/textfile tables, but that is just the luck of the path taken by the code. Inspection of my PG metastore shows all the CLOB fields suffering from this issue.

      Attachments

        1. HIVE-16667.2.patch
          2 kB
          Naveen Gangam
        2. HIVE-16667.3.patch
          2 kB
          Naveen Gangam
        3. HIVE-16667.patch
          3 kB
          Naveen Gangam
        4. HiveCLIOutput.txt
          98 kB
          Naveen Gangam
        5. PostgresDBOutput.txt
          3 kB
          Naveen Gangam

        Issue Links

          Activity

            People

              ngangam Naveen Gangam
              rusanu Remus Rusanu
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: