Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13345 S3Guard: Improved Consistency for S3A
  3. HADOOP-14423

s3guard will set file length to -1 on a putObjectDirect(stream, -1) call

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.0.0-alpha4
    • HADOOP-13345
    • fs/s3
    • None

    Description

      You can pass a negative number into S3AFileSystem.putObjectDirect, which means "put until the end of the stream". S3guard has been using this len argument: it needs to be using the actual number of bytes uploaded. Also relevant with client side encryption, when the amount of data put > the amount of data in the file or stream.

      Noted in the committer branch after I added some more assertions, I've changed it there so making changes to S3AFS.putObjectDirect to pull the content length to pass in to finishedWrite() from the PutObjectResult instead. This can be picked into the s3guard branch

      Attachments

        1. HADOOP-14423-HADOOP-13345-002.patch
          5 kB
          Steve Loughran
        2. HADOOP-14423-HADOOP-13345-001.patch
          5 kB
          Steve Loughran

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            stevel@apache.org Steve Loughran
            stevel@apache.org Steve Loughran
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment