Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-22054

Avoid recursive listing to check if a directory is empty

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.13.0, 1.2.0, 2.1.0, 2.3.5, 3.1.1
    • 4.0.0-alpha-1
    • Metastore
    • None

    Description

      During drop partition on a managed table, first we delete the directory corresponding to the partition. After that we recursively delete the parent directory as well if parent directory becomes empty. To do this emptiness check, we call Warehouse::getContentSummary(), which in turn recursively check all files and subdirectories. This is a costly operation when a directory has a lot of files or subdirectories. This overhead is even more prominent for cloud based file systems like s3. And for emptiness check, this is unnecessary too.

      This is recursive listing was introduced as part of HIVE-5220. Code snippet for reference :

      // Warehouse.java
      public boolean isEmpty(Path path) throws IOException, MetaException {
        ContentSummary contents = getFs(path).getContentSummary(path);
        if (contents != null && contents.getFileCount() == 0 && contents.getDirectoryCount() == 1) {
          return true;
        }
        return false;
      }
      
      // HiveMetaStore.java
      private void deleteParentRecursive(Path parent, int depth, boolean mustPurge, boolean needRecycle)
        throws IOException, MetaException {
        if (depth > 0 && parent != null && wh.isWritable(parent)) {
          if (wh.isDir(parent) && wh.isEmpty(parent)) {
            wh.deleteDir(parent, true, mustPurge, needRecycle);
          }
          deleteParentRecursive(parent.getParent(), depth - 1, mustPurge, needRecycle);
        }
      }
      
      // Note: FileSystem::getContentSummary() performs a recursive listing.

      Attachments

        1. HIVE-22054.2.patch
          2 kB
          Prabhas Kumar Samanta
        2. HIVE-22054.patch
          2 kB
          Prabhas Kumar Samanta

        Issue Links

          Activity

            People

              prabhas Prabhas Kumar Samanta
              prabhas Prabhas Kumar Samanta
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: