Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7060

Cherry Pick PathOutputCommitter class/factory to branch-3.0

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.0.0, 2.10.0
    • 3.0.3
    • None
    • None

    Description

      It's easier for downstream apps like Spark to pick up the new PathOutputCommitter superclass if it is there on 2.10+, even if the S3A committer isn't there.

      Adding the interface & binding stuff of MAPREDUCE-6956 allows for third party committers to be deployed.

      I'm not proposing a backport of the HADOOP-13786 committer: that's Java 8, S3Guard, etc. Too traumatic. All I want here is to allow downstream code to be able to pick up the new interface and so be able to support it and other store committers when available

      Attachments

        1. MAPREDUCE-7060-branch-3.0-001.patch
          44 kB
          Steve Loughran
        2. MAPREDUCE-7060-branch-3.0-002.patch
          46 kB
          Steve Loughran

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-6956 FileOutputCommitter to gain abstract superclass PathOutputCommitter

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          relates to

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-13786 Add S3A committers for zero-rename commits to S3 endpoints

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: