Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-11634

Support partition pruning for IN(STRUCT(partcol, nonpartcol..)...)

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.0.0
    • CBO

    Description

      Currently, we do not support partition pruning for the following scenario

      create table pcr_t1 (key int, value string) partitioned by (ds string);
      insert overwrite table pcr_t1 partition (ds='2000-04-08') select * from src where key < 20 order by key;
      insert overwrite table pcr_t1 partition (ds='2000-04-09') select * from src where key < 20 order by key;
      insert overwrite table pcr_t1 partition (ds='2000-04-10') select * from src where key < 20 order by key;
      explain extended select ds from pcr_t1 where struct(ds, key) in (struct('2000-04-08',1), struct('2000-04-09',2));
      

      If we run the above query, we see that all the partitions of table pcr_t1 are present in the filter predicate where as we can prune partition (ds='2000-04-10').

      The optimization is to rewrite the above query into the following.

      explain extended select ds from pcr_t1 where  (struct(ds)) IN (struct('2000-04-08'), struct('2000-04-09')) and  struct(ds, key) in (struct('2000-04-08',1), struct('2000-04-09',2));
      

      The predicate (struct(ds)) IN (struct('2000-04-08'), struct('2000-04-09')) is used by partition pruner to prune the columns which otherwise will not be pruned.
      This is an extension of the idea presented in HIVE-11573.

      Attachments

        1. HIVE-11634.1.patch
          13 kB
          Hari Sankar Sivarama Subramaniyan
        2. HIVE-11634.2.patch
          22 kB
          Hari Sankar Sivarama Subramaniyan
        3. HIVE-11634.3.patch
          67 kB
          Hari Sankar Sivarama Subramaniyan
        4. HIVE-11634.4.patch
          75 kB
          Hari Sankar Sivarama Subramaniyan
        5. HIVE-11634.5.patch
          78 kB
          Hari Sankar Sivarama Subramaniyan
        6. HIVE-11634.6.patch
          84 kB
          Hari Sankar Sivarama Subramaniyan
        7. HIVE-11634.7.patch
          85 kB
          Hari Sankar Sivarama Subramaniyan
        8. HIVE-11634.8.patch
          85 kB
          Hari Sankar Sivarama Subramaniyan
        9. HIVE-11634.9.patch
          90 kB
          Hari Sankar Sivarama Subramaniyan
        10. HIVE-11634.91.patch
          112 kB
          Hari Sankar Sivarama Subramaniyan
        11. HIVE-11634.92.patch
          110 kB
          Hari Sankar Sivarama Subramaniyan
        12. HIVE-11634.93.patch
          104 kB
          Hari Sankar Sivarama Subramaniyan
        13. HIVE-11634.94.patch
          105 kB
          Hari Sankar Sivarama Subramaniyan
        14. HIVE-11634.95.patch
          105 kB
          Hari Sankar Sivarama Subramaniyan
        15. HIVE-11634.96.patch
          105 kB
          Hari Sankar Sivarama Subramaniyan
        16. HIVE-11634.97.patch
          104 kB
          Hari Sankar Sivarama Subramaniyan
        17. HIVE-11634.98.patch
          104 kB
          Hari Sankar Sivarama Subramaniyan
        18. HIVE-11634.99.patch
          116 kB
          Hari Sankar Sivarama Subramaniyan
        19. HIVE-11634.990.patch
          134 kB
          Hari Sankar Sivarama Subramaniyan
        20. HIVE-11634.991.patch
          132 kB
          Hari Sankar Sivarama Subramaniyan
        21. HIVE-11634.992.patch
          155 kB
          Hari Sankar Sivarama Subramaniyan
        22. HIVE-11634.993.patch
          132 kB
          Hari Sankar Sivarama Subramaniyan
        23. HIVE-11634.994.patch
          149 kB
          Hari Sankar Sivarama Subramaniyan
        24. HIVE-11634.patch
          151 kB
          Hari Sankar Sivarama Subramaniyan
        25. HIVE-11634.995.patch
          151 kB
          Hari Sankar Sivarama Subramaniyan

        Issue Links

          Activity

            People

              hsubramaniyan Hari Sankar Sivarama Subramaniyan
              hsubramaniyan Hari Sankar Sivarama Subramaniyan
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: