Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-18477 Über-jira: S3A Hadoop 3.3.9 features
  3. HADOOP-17415

Use S3 content-range header to update length of an object during reads

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.3.0
    • None
    • fs/s3

    Description

      As part of all the openFile work, knowing full length of an object allows for a HEAD to be skipped. But: code knowing only the splits don't know the final length of the file.

      If the content-range header is used, then as soon as a single GET is initiated against an object, if the field is returned then we can update the length of the S3A stream to its real/final length

      Also: when any input stream fails with an EOF exception, we can distinguish stream-interrupted from "no, too far"

      Attachments

        Issue Links

          Activity

            People

              monthonk Monthon Klongklaew
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 50m
                  50m