Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28983

Can not pruning hive partitions by Or predicate

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Duplicate
    • 2.4.4
    • None
    • SQL
    • None

    Description

      Assume the hive partition table has p1/p2 partition column and c1 non-partition column,

      the below where clause can not prunning partitions in sparksql, but we can prunning by "p1 = 'a' or p2 = 'b'" in principal.

      where
      (p1 = 'a' and c1 = 1)
      or
      (p2 = 'b' and c1 = 2)

       

      The problem is in HiveTableScans#apply(plan: LogicalPlan),  when the predicate has any child that contains non-partition column, it will make a outer FilterExec and empty predicate in HiveTableScanExec.

       

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              924060929 huajian lan
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: