Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-16202

Enhance openFile() for better read performance against object stores

    XMLWordPrintableJSON

Details

    Description

      The openFile() builder API lets us add new options when reading a file

      Add an option "fs.s3a.open.option.length" which takes a long and allows the length of the file to be declared. If set, no check for the existence of the file is issued when opening the file

      Also: withFileStatus() to take any FileStatus implementation, rather than only S3AFileStatus -and not check that the path matches the path being opened. Needed to support viewFS-style wrapping and mounting.

      and Adopt where appropriate to stop clusters with S3A reads switched to random IO from killing download/localization

      • fs shell copyToLocal
      • distcp
      • IOUtils.copy

      this patch MUST be followed by HADOOP-18410

      Attachments

        Issue Links

          breaks

          Sub-task - The sub-task of the issue HADOOP-18410 S3AInputStream.unbuffer() async drain not releasing http connections

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-7391 TestLocalDistributedCacheManager failing after HADOOP-16202

          • Major - Major loss of function.
          • Resolved
          causes

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-18724 Open file fails with NumberFormatException for S3AFileSystem

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          contains

          Sub-task - The sub-task of the issue HADOOP-17882 distcp to use openFile() with sequential IO; ranges of reads

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-17883 openFile() to add standard options for read policies, start/end ranges

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-17884 FsShell to use openFile(sequential) for reads

          • Major - Major loss of function.
          • Resolved
          incorporates

          Sub-task - The sub-task of the issue HADOOP-17896 ABFS: Stabilize openFile withStatus

          • Major - Major loss of function.
          • Resolved
          is depended upon by

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-18179 Boost S3A Stream Read Performance

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. YARN-10444 Node Manager to use openFile() with whole-file read policy for localizing files.

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is duplicated by

          Sub-task - The sub-task of the issue HADOOP-16387 S3A openFile() options to allow etag/version to be set

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Reopened
          is related to

          Sub-task - The sub-task of the issue HADOOP-15625 S3A input stream to use etags/version number to detect changed source files

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-17682 ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-16090 S3A Client to add explicit support for versioned stores

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-18944 s3a openfile is using readahead rather than async drain threshold to calculate drain threshold

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-14478 Add libhdfs APIs for openFile

          • Major - Major loss of function.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue HADOOP-17038 Support disabling buffered reads in ABFS positional reads

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-16759 Filesystem openFile() builder to take a FileStatus param

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. TEZ-4522 Use OpenFile where FileStatus is available.

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #2168

          Web Link GitHub Pull Request #2584

          Web Link GitHub Pull Request #4238

          Web Link https://github.com/apache/hadoop/pull/2046

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 23h 10m
                  23h 10m