Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11694 Über-jira: S3a phase II: robustness, scale and performance
  3. HADOOP-13106

if fs.s3a.block.size option == 0, use partition size option for blocksize

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Minor
    • Resolution: Won't Fix
    • 2.7.2
    • None
    • fs/s3
    • None

    Description

      Input perf is best if blocksize matches the partition size used to upload the data.

      There's currently no way of knowing this —but S3A does know the partition size to use when uploading data.

      I propose picking that up as the blocksize if the configured value of fs.s3a.block.size is zero. This value is utterly illegal today (more precisely, it will break any app using it to calculate splits), even when there is no check for its range

      Attachments

        Activity

          People

            stevel@apache.org Steve Loughran
            stevel@apache.org Steve Loughran
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: