Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.1
    • 3.3.1, 3.4.0
    • fs/azure
    • None

    Description

      Initially changes were made as part of this PR to handle idempotency including rename operation with the understanding that last modified time gets updated. But that assumption was wrong and the rename idempotency handling has since evolved.

      For a job clean up, if the Manifest Committer in below Jira is used, then rename idempotency works using the previously fetched etag :

      HADOOP-18163 hadoop-azure support for the Manifest Committer of MAPREDUCE-7341 - ASF JIRA (apache.org)

       

      A part of the commit tracked under current Jira to handle DELETE idempotency is still relevant.

      A means to handle idempotency between driver and backend inherently is being worked upon.

      – Older notes

      Currently when a PUT or POST operation timeouts and the server has already successfully executed the operation, there is no check in driver to see if the operation did succeed or not and just retries the same operation again. This can cause driver to through invalid user errors.

       

      Sample scenario:

      1. Rename request times out. Though server has successfully executed the operation.
      2. Driver retries rename and get source not found error.

      In the scenario, driver needs to check if rename is being retried and success if source if not found, but destination is present.

       

      Attachments

        Issue Links

          Activity

            People

              snvijaya Sneha Vijayarajan
              snvijaya Sneha Vijayarajan
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: