Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-26000

DirectSQL to prune partitions fails with postgres backend for Skewed-Partition tables

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • None
    • None

    Description

       

       

      2022-03-02 20:37:56,421 INFO  org.apache.hadoop.hive.metastore.PartFilterExprUtil: [pool-6-thread-200]: Unable to make the expression tree from expression string [((ds = '2008-04-08') and (UDFToDouble(hr) = 11.0D))]Error parsing partition filter; lexer error: null; exception NoViableAltException(24@[])
      2022-03-02 20:37:56,593 WARN  org.apache.hadoop.hive.metastore.ObjectStore: [pool-6-thread-200]: Falling back to ORM path due to direct SQL failure (this is not an error): Error executing SQL query "select "SKEWED_COL_VALUE_LOC_MAP"."SD_ID", "SKEWED_STRING_LIST_VALUES".STRING_LIST_ID, "SKEWED_COL_VALUE_LOC_MAP"."LOCATION", "SKEWED_STRING_LIST_VALUES"."STRING_LIST_VALUE" from "SKEWED_COL_VALUE_LOC_MAP"  left outer join "SKEWED_STRING_LIST_VALUES" on "SKEWED_COL_VALUE_LOC_MAP"."STRING_LIST_ID_KID" = "SKEWED_STRING_LIST_VALUES"."STRING_LIST_ID" where "SKEWED_COL_VALUE_LOC_MAP"."SD_ID" in (51010)  and "SKEWED_COL_VALUE_LOC_MAP"."STRING_LIST_ID_KID" is not null order by "SKEWED_COL_VALUE_LOC_MAP"."SD_ID" asc,  "SKEWED_STRING_LIST_VALUES"."STRING_LIST_ID" asc,  "SKEWED_STRING_LIST_VALUES"."INTEGER_IDX" asc". at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543) at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391) at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:216) at org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.loopJoinOrderedResult(MetastoreDirectSqlUtils.java:131) at org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.loopJoinOrderedResult(MetastoreDirectSqlUtils.java:109) at org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.setSkewedColLocationMaps(MetastoreDirectSqlUtils.java:414) at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsFromPartitionIds(MetaStoreDirectSql.java:967) at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsFromPartitionIds(MetaStoreDirectSql.java:788) at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.access$300(MetaStoreDirectSql.java:117) at org.apache.hadoop.hive.metastore.MetaStoreDirectSql$1.run(MetaStoreDirectSql.java:530) at org.apache.hadoop.hive.metastore.Batchable.runBatched(Batchable.java:73) at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilter(MetaStoreDirectSql.java:521) at org.apache.hadoop.hive.metastore.ObjectStore$10.getSqlResult(ObjectStore.java:3722);
       Caused by: ERROR: column SKEWED_STRING_LIST_VALUES.string_list_id does not exist
      

       

      Attachments

        Issue Links

          Activity

            People

              nareshpr Naresh P R
              nareshpr Naresh P R
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h
                  1h