Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-17039 Implement optimization rewritings that rely on database SQL constraints
  3. HIVE-17043

Remove non unique columns from group by keys if not referenced later

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 4.0.0-alpha-1
    • Logical Optimizer
    • None

    Description

      Group by keys may be a mix of unique (or primary) keys and regular columns. In such cases presence of regular column won't alter cardinality of groups. So, if regular columns are not referenced later, they can be dropped from group by keys. Depending on operator tree may result in those columns not being read at all from disk in best case. In worst case, we will avoid shuffling and sorting regular columns from mapper to reducer, which still could be substantial CPU and network savings.

      Attachments

        1. HIVE-17043.1.patch
          84 kB
          Vineet Garg
        2. HIVE-17043.2.patch
          108 kB
          Vineet Garg
        3. HIVE-17043.3.patch
          131 kB
          Vineet Garg
        4. HIVE-17043.4.patch
          113 kB
          Vineet Garg
        5. HIVE-17043.5.patch
          127 kB
          Vineet Garg
        6. HIVE-17043.6.patch
          132 kB
          Vineet Garg
        7. HIVE-17043.7.patch
          136 kB
          Vineet Garg
        8. HIVE-17043.8.patch
          137 kB
          Vineet Garg
        9. HIVE-17043.9.patch
          136 kB
          Vineet Garg
        10. HIVE-17043.10.patch
          139 kB
          Vineet Garg
        11. HIVE-17043.11.patch
          141 kB
          Vineet Garg
        12. HIVE-17043.12.patch
          158 kB
          Vineet Garg
        13. HIVE-17043.13.patch
          158 kB
          Vineet Garg
        14. HIVE-17043.14.patch
          143 kB
          Vineet Garg
        15. HIVE-17043.15.patch
          143 kB
          Vineet Garg
        16. HIVE-17043.16.patch
          143 kB
          Vineet Garg

        Issue Links

          Activity

            People

              vgarg Vineet Garg
              ashutoshc Ashutosh Chauhan
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: