Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15620 Über-jira: S3A phase VI: Hadoop 3.3 features
  3. HADOOP-16823

Large DeleteObject requests are their own Thundering Herd

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.1
    • 3.3.0
    • fs/s3
    • None
    • Hide
      The page size for bulk delete operations has been reduced from 1000 to 250 to reduce the likelihood of overloading an S3 partition, especially because the retry policy on throttling is simply to try again.

      The page size can be set in "fs.s3a.bulk.delete.page.size"

      There is also an option to control whether or not the AWS client retries requests, or whether it is handled exclusively in the S3A code. This option "fs.s3a.experimental.aws.s3.throttling" is true by default. If set to false: everything is handled in the S3A client. While this means that metrics may be more accurate, it may mean that throttling failures in helper threads of the AWS SDK (especially those used in copy/rename) may not be handled properly. This is experimental, and should be left at "true" except when seeking more detail about throttling rates.

      Show
      The page size for bulk delete operations has been reduced from 1000 to 250 to reduce the likelihood of overloading an S3 partition, especially because the retry policy on throttling is simply to try again. The page size can be set in "fs.s3a.bulk.delete.page.size" There is also an option to control whether or not the AWS client retries requests, or whether it is handled exclusively in the S3A code. This option "fs.s3a.experimental.aws.s3.throttling" is true by default. If set to false: everything is handled in the S3A client. While this means that metrics may be more accurate, it may mean that throttling failures in helper threads of the AWS SDK (especially those used in copy/rename) may not be handled properly. This is experimental, and should be left at "true" except when seeking more detail about throttling rates.

    Description

      Currently AWS S3 throttling is initially handled in the AWS SDK, only reaching the S3 client code after it has given up.

      This means we don't always directly observe when throttling is taking place.

      Proposed:

      • disable throttling retries in the AWS client library
      • add a quantile for the S3 throttle events, as DDB has
      • isolate counters of s3 and DDB throttle events to classify issues better

      Because we are taking over the AWS retries, we will need to expand the initial delay en retries and the number of retries we should support before giving up.

      Also: should we log throttling events? It could be useful but there is a risk of logs overloading especially if many threads in the same process were triggering the problem.

      Proposed: log at debug.

      Note: if S3 bucket logging is enabled then throttling events will be recorded as 503 responses in the logs. If the hadoop version contains the audit logging of HADOOP-17511, this can be used to identify operations/jobs/users which are triggering problems.

      Attachments

        Issue Links

          is related to

          Sub-task - The sub-task of the issue HADOOP-13811 s3a: getFileStatus fails with com.amazonaws.AmazonClientException: Failed to sanitize XML document destined for handler class

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-17935 Spark job stuck in S3A StagingCommitter::setupJob

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #1814

          Web Link GitHub Pull Request #1826

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: