Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-19092 ABFS phase 4: post Hadoop 3.4.0 features
  3. HADOOP-18759

[ABFS][Backoff-Optimization] Have a Static retry policy for connection timeout failures

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.3.4
    • 3.5.0
    • fs/azure
    • None

    Description

      Today when a request fails with connection timeout, it falls back into the loop for exponential retry. Unlike Azure Storage, there are no guarantees of success on exponentially retried request or recommendations for ideal retry policies for Azure network or any other general failures. Faster failure and retry might be more beneficial for such generic connection timeout failures. 

      This PR introduces a new Static Retry Policy which will currently be used only for Connection Timeout failures. It means all the requests failing with Connection Timeout errors will be retried after a constant retry(sleep) interval independent of how many times that request has failed. Max Retry Count check will still be in place.

      Following Configurations will be introduced in the change:

      1. "fs.azure.static.retry.for.connection.timeout.enabled" - default: true, true: static retry will be used for CT, false: Exponential retry will be used.
      2. "fs.azure.static.retry.interval" - default: 1000ms.

      This also introduces a new field in x-ms-client-request-id only for the requests that are being retried after connection timeout failure. New filed will tell what retry policy was used to get the sleep interval before making this request.
      Header "x-ms-client-request-id " right now has only the retryCount and retryReason this particular API call is. For ex:  :eb06d8f6-5693-461b-b63c-5858fa7655e6:29cb0d19-2b68-4409-bc35-cb7160b90dd8:::CF:1_CT.

      Moving ahead for retryReason "CT" it will have retry policy abbreviation as well.
      For ex:  :eb06d8f6-5693-461b-b63c-5858fa7655e6:29cb0d19-2b68-4409-bc35-cb7160b90dd8:::CF:1_CT_E.

      Attachments

        Activity

          People

            anujmodi2021 Anuj Modi
            anujmodi2021 Anuj Modi
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: