Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-16736

remove redundant FileSystem status checks calls from Spark codebase

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.0.0
    • 2.1.0
    • Spark Core
    • None

    Description

      The Hadoop FileSystem.exists() and FileSystem.isDirectory() calls are wrappers around FileSystem.getStatus(), —the latter putting load on an HDFS NN, and very, very slow against object stores.

      1. if these calls are followed by any getStatus() calls then they can be eliminated by careful merging and pulling out the catching of {FileNotFoundException}} from the exists() call to the spark code.
      1. Any sequence of exists + delete can be optimised by removing the exists check, relying on FileSystem.delete() to be a no-op if the destination path is not present. That's a tested requirement of all Hadoop compatible FS and object stores.

      Attachments

        Issue Links

          is depended upon by

          Sub-task - The sub-task of the issue HADOOP-13525 Optimize uses of FS operations in the ASF analysis frameworks and libraries

          • Major - Major loss of function.
          • Resolved
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-13427 Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-15192 S3A listStatus excessively slow -hurts Spark job partitioning

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-13321 Deprecate FileSystem APIs that promote inefficient call patterns.

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-14323 Reduce number of FS permissions and redundant FS operations

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          links to

          Pull request #14371 [Github] Pull Request #14371 (steveloughran)

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: