Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14764 Über-jira adl:// Azure Data Lake Phase II: Performance, Resilience and Testing
  3. HADOOP-12876

[Azure Data Lake] Support for process level FileStatus cache to optimize GetFileStatus frequent operations

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • None
    • None
    • fs, fs/adl, tools
    • None

    Description

      Add support to cache GetFileStatus and ListStatus response locally for limited period of time. Local cache for limited period of time would optimize number of calls for GetFileStatus operation.
      One of the example where local limited period cache would be useful - terasort ListStatus on input directory follows with GetFileStatus operation on each file within directory. For 2048 input files in a directory would save 2048 GetFileStatus calls during start up (Using the ListStatus response to cache FileStatus instances).

      Attachments

        Issue Links

          depends upon

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-15038 Abstract MetadataStore in S3Guard into a common module.

          • Major - Major loss of function.
          • Patch Available
          is related to

          Sub-task - The sub-task of the issue HADOOP-13452 S3Guard: Implement access policy for intra-client consistency with in-memory metadata store.

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-13321 Deprecate FileSystem APIs that promote inefficient call patterns.

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              vishwajeet.dusane Vishwajeet Dusane
              vishwajeet.dusane Vishwajeet Dusane
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: