Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-18067 Über-jira: S3A Hadoop 3.3.5 features
  3. HADOOP-18339

S3A storage class option only picked up when buffering writes to disk

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      when you switch s3a output stream buffering to heap or byte buffer, the storage class option isn't added to the put request

      
  <property>
    <name>fs.s3a.fast.upload.buffer</name>
    <value>bytebuffer</value>
  </property>

      and the ITestS3AStorageClass tests fail.

      
java.lang.AssertionError: [Storage class of object s3a://stevel-london/test/testCreateAndCopyObjectWithStorageClassGlacier/file1] 
Expecting:
 <null>
to be equal to:
 <"glacier">
ignoring case considerations

	at org.apache.hadoop.fs.s3a.ITestS3AStorageClass.assertObjectHasStorageClass(ITestS3AStorageClass.java:215)
	at org.apache.hadoop.fs.s3a.ITestS3AStorageClass.testCreateAndCopyObjectWithStorageClassGlacier(ITestS3AStorageClass.java:129)

      we noticed this in a code review; the request factory only sets the option when the source is a file, not memory.

      proposed: parameterize the test suite on disk/byte buffer, then fix

      Attachments

        Issue Links

          is caused by

          Sub-task - The sub-task of the issue HADOOP-12020 Support configuration of different S3 storage classes

          • Major - Major loss of function.
          • Resolved
          is depended upon by

          Sub-task - The sub-task of the issue HADOOP-16259 Distcp to set S3 Storage Class

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Sub-task - The sub-task of the issue HADOOP-12020 Support configuration of different S3 storage classes

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #4669

          Activity

            People

              monthonk Monthon Klongklaew
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m