Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-17833

Improve Magic Committer Performance

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.3.1
    • 3.3.5
    • fs/s3
    • S3A filesytem's createFile() operation supports an option to disable all safety checks when creating a file. Consult the documentation and use with care

    Description

      Magic committer tasks can be slow because every file created with overwrite=false triggers a HEAD (verify there's no file) and a LIST (that there's no dir). And because of delayed manifestations, it may not behave as expected.

      ParquetOutputFormat is one example of a library which does this.

      we could fix parquet to use overwrite=true, but (a) there may be surprises in other uses (b) it'd still leave the list and (c) do nothing for other formats call

      Proposed: createFile() under a magic path to skip all probes for file/dir at end of path

      Only a single task attempt Will be writing to that directory and it should know what it is doing. If there is conflicting file names and parts across tasks that won't even get picked up at this point. Oh and none of the committers ever check for this: you'll get the last file manifested (s3a) or renamed (file)

      If we skip the checks we will save 2 HTTP requests/file.

      Attachments

        Issue Links

          blocks

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-18281 Tune S3A storage class support

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open
          breaks

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-18402 S3A committer NPE in spark job abort

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          causes

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-18757 S3A Committer only finalizes the commits in a single thread

          • Major - Major loss of function.
          • Resolved
          contains

          Sub-task - The sub-task of the issue HADOOP-15460 S3A FS to add "fs.s3a.create.performance" to the builder file creation option set

          • Major - Major loss of function.
          • Resolved
          depends upon

          New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-7341 Add a task-manifest output committer for Azure and GCS

          • Major - Major loss of function.
          • Resolved
          incorporates

          Sub-task - The sub-task of the issue HADOOP-16017 Add some S3A-specific create file options

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Sub-task - The sub-task of the issue HADOOP-17584 s3a magic committer may commit more data

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-18162 hadoop-common enhancements for the Manifest Committer of MAPREDUCE-7341

          • Major - Major loss of function.
          • Resolved

          Wish - General wishlist item. HADOOP-18568 Magic Committer optional clean up

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-17935 Spark job stuck in S3A StagingCommitter::setupJob

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #3289

          Web Link GitHub Pull Request #4470

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 14h
                  14h