Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-20284

In strict mode, if constant propagation is enable, the partition filter may be folded before partition pruner lead to error "No partition predicate for Alias"

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Patch Available
    • Trivial
    • Resolution: Unresolved
    • 1.2.1, 2.3.3
    • None
    • CLI
    • None

    Description

      In strict mode and the hive.optimize.constant.propagation is set to true, the following sql will failed:

      hive> desc employee_part;
      OK
      col_name	data_type	comment
      eid                 	int
      name                	string
      dept                	string
      year                	string
      month               	string
      
      # Partition Information
      # col_name            	data_type           	comment
      
      year                	string
      month               	string
      Time taken: 0.564 seconds, Fetched: 11 row(s)
      hive> set hive.mapred.mode=strict;
      hive> select * from employee_part where false and concat(year,month)='201807';
      FAILED: SemanticException Queries against partitioned tables without a partition filter are disabled for safety reasons. If you know what you are doing, please sethive.strict.checks.large.query to false and that hive.mapred.mode is not set to 'strict' to proceed. Note that if you may get errors or incorrect results if you make a mistake while using some of the unsafe features. No partition predicatie for Alias "employee_part" Table "employee_part"
      

      The above error message is confusing because the expression concat(year,month)='201807' is a partition filter。

      The reason is during logic optimization, the ConstantPropagate optimizer is running before partitionPruner optimizer, when found a express like 'false and concat(year,month)=xxxx', the express is replaced with 'fasle' , and the partition filter is dropped. So the PartitionPruner can not get the partition filter.

      Users can remove the constant express that always has true/false values to work around.

      When views used, if some columns are constant values, users will be confusing.

      So we should add some more message in the error msg returned.

      Attachments

        1. HIVE-20284.patch
          8 kB
          Hui Huang
        2. HIVE-20284.1.patch
          15 kB
          Hui Huang
        3. HIVE-20284.2.patch
          13 kB
          Hui Huang
        4. HIVE-20284.3.patch
          15 kB
          Hui Huang

        Activity

          People

            Unassigned Unassigned
            BIGrey Hui Huang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: