Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.6.0
    • 2.7.0
    • fs/s3
    • None

    Description

      Currently s3a buffers files on disk(s) before uploading. This JIRA investigates adding a memory-based upload implementation.

      The motivation is evidently performance: this would be beneficial for users with high network bandwidth to S3 (EC2?) or users that run Hadoop directly on an S3-compatible object store (FYI: my contributions are made in name of Amplidata).

      Attachments

        1. design-comments.pdf
          44 kB
          Thomas Demoor
        2. HADOOP-11183.001.patch
          25 kB
          Thomas Demoor
        3. HADOOP-11183.002.patch
          26 kB
          Thomas Demoor
        4. HADOOP-11183.003.patch
          22 kB
          Thomas Demoor
        5. HADOOP-11183-004.patch
          22 kB
          Thomas Demoor
        6. HADOOP-11183-005.patch
          26 kB
          Thomas Demoor
        7. HADOOP-11183-006.patch
          28 kB
          Thomas Demoor
        8. HADOOP-11183-007.patch
          29 kB
          Thomas Demoor
        9. HADOOP-11183-008.patch
          30 kB
          Steve Loughran
        10. HADOOP-11183-009.patch
          30 kB
          Thomas Demoor
        11. HADOOP-11183-010.patch
          30 kB
          Thomas Demoor

        Issue Links

          Activity

            People

              thodemoor Thomas Demoor
              thodemoor Thomas Demoor
              Votes:
              1 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: