Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-27595

Improve efficiency in the filtering hooks

    XMLWordPrintableJSON

Details

    Description

      https://github.com/apache/hive/blob/a406d6d4417277e45b93f1733bed5201afdee29b/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/metastore/HiveMetaStoreAuthorizer.java#L353-L377

      In case where the tableList has large amounts of tables (tested with 200k in my case), the hivePrivilegedObjects could just as big. So both these lists are 200k.

      Essentially. the code is trying to return a subset of tableList collection that matches the objects returned in hivePrivilegedObjects. This results in a N*N iteration that causes bad performance. (in my case, the HMS client timeout expired and show tables failed).

      This code needs to be optimized for performance.

      we have a similar problem in this code as well.
      ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/AuthorizationMetaStoreFilterHook.java

      Attachments

        Issue Links

          Activity

            People

              henrib Henri Biestro
              ngangam Naveen Gangam
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: