Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-36137

HiveShim always fallback to getAllPartitionsOf regardless of whether directSQL is enabled in remote HMS

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.0
    • 3.3.0
    • SQL
    • None

    Description

      At the moment getPartitionsByFilter in Hive shim only fallback to use getAllPartitionsOf when hive.metastore.try.direct.sql is enabled in the remote HMS. However, in certain cases the remote HMS will fallback to use ORM (which only support string type for partition columns) to query the underlying RDBMS even if this config is set to true, and Spark will not be able to recover from the error and will just fail the query.

      For instance, we encountered this bug HIVE-21497 in HMS running Hive 3.1.2, and Spark was not able to pushdown filter for date column.

      Attachments

        Issue Links

          Activity

            People

              csun Chao Sun
              csun Chao Sun
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: