Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-27984

Support backward compatibility of hms thrift struct about column stats

    XMLWordPrintableJSON

Details

    Description

      We had some discussions about column stats thrift api isssue before: https://www.mail-archive.com/dev@hive.apache.org/msg144294.html ,

      and i also did some attempt to fix this issue from TrinoDB side when i playing TrinoDB&Hive4 https://github.com/trinodb/trino-hive-apache/pull/43 .

      Recently, i tried to use starrocks(https://github.com/StarRocks/starrocks) which uses a hms client compatible with Hive3 to query Hive4, and i also encountered this issue(https://github.com/StarRocks/starrocks/issues/38620), 

       

      Caused by: org.apache.thrift.TApplicationException: Required field 'engine' is unset! Struct:TableStatsRequest(dbName:testdb, tblName:testsr, colNames:[id], engine:null)
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) ~[libthrift-0.13.0.jar:0.13.0]
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_statistics_req(ThriftHiveMetastore.java:4161) ~[hive-apache-3.1.2-13.jar:?]
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_statistics_req(ThriftHiveMetastore.java:4148) ~[hive-apache-3.1.2-13.jar:?]
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTableColumnStatistics(HiveMetaStoreClient.java:735) ~[starrocks-fe.jar:?]
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTableColumnStatistics(HiveMetaStoreClient.java:727) ~[starrocks-fe.jar:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_221]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_221]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_221]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_221]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:208) ~[hive-apache-3.1.2-13.jar:?]
        at com.sun.proxy.$Proxy32.getTableColumnStatistics(Unknown Source) ~[?:?]

       

      In addition, i also found similar ticket in Impala https://issues.apache.org/jira/browse/IMPALA-12539 .

      That is to say, if other components(Impala&TrinoDB&Starrocks) want to use hive3 hms client to get the column stats to make CBO, they will get the execption from Hive4. I think it is necessary to support  backward compatibility to make other compontents work well with hive4(hms4).

      Attachments

        Issue Links

          is related to

          Task - A task that needs to be done. IMPALA-12539 hive-4 compatibility with Impala

          • Major - Major loss of function.
          • Open
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-22046 Differentiate among column stats computed by different engines

          • Major - Major loss of function.
          • Closed
          links to

          Web Link GitHub Pull Request #4984

          Activity

            People

              zhangbutao Butao Zhang
              zhangbutao Butao Zhang
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: