Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-22312

MapJoinCounterHook doesnot work for tez

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: All Versions
    • Fix Version/s: None
    • Component/s: Hive
    • Labels:
      None

      Description

      In HIVE-1792, MapJoinCounterHook hook was added to track joins that get converted to map join. This hook gets List of Tasks from hookContext and check Tag associated with each task. For mr, we create Conditional tasks in case of joins and add tags for respective join conversions. This does not work in case of tez as we only create TezTask(no Conditional Task is created) which can handle multiple joins in contrast to one Conditional Task per join in mr.
      The current approach will fail even if we add tag to TezTask as it can have multiple joins of the same type which will require counter.
      One possible solution for tez, is to parse query-plan after query completion which we get from hookContext to get workGraph. Using workGraph, we can walk through Operator Tree to find join conversions.

       If this approach looks good, I can raise Pull Request
      cc Ashutosh Chauhan Jesus Camacho Rodriguez Pengcheng Xiong 

        Attachments

        1. HIVE-22312.01.patch
          13 kB
          Pulkit Sharma
        2. HIVE-22312.02.patch
          13 kB
          Pulkit Sharma
        3. HIVE-22312.03.patch
          13 kB
          Pulkit Sharma

          Issue Links

            Activity

              People

              • Assignee:
                pulkits Pulkit Sharma
                Reporter:
                pulkits Pulkit Sharma
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: