Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-9826

Fix exception handling if one Datanode is not available (Ratis)

    XMLWordPrintableJSON

Details

    Description

      When a key is uploading by XcieverClientRatis, and some datanode becomes unavailable, it is expected that client should request new pipeline to retry upload.

      In fact, before that client tries to repeat commit check with MAJORITY_COMMITTED replication level, which cannot be successful as at that moment pipeline is already closed.

      XceiverClientRatis has method watchForCommit(long index), which contains exception check

       

      if (t instanceof GroupMismatchException) {
  throw e;
}

      GroupMismatchException throws by Ratis client exactly when some datanode is not available and further key upload is not available for current pipeline.

      But this check does not work as 

      Throwable t = HddsClientUtils.checkForException(e);

       does not unwrap exception completely.

      The idea is fix lookup of nested exceptions to find proper one. This improves failover latency by 15 seconds approximately.

      Attachments

        Issue Links

          is caused by

          Bug - A problem which impairs or prevents the functions of the product. HDDS-2280 HddsUtils#CheckForException should not return null in case the ratis exception cause is not set

          • Major - Major loss of function.
          • Resolved
          is related to

          Sub-task - The sub-task of the issue HDDS-10788 Intermittent failure in testWatchForCommitForRetryfailure

          • Major - Major loss of function.
          • In Progress

          Bug - A problem which impairs or prevents the functions of the product. HDDS-1395 Key write fails with BlockOutputStream has been closed exception

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDDS-9823 Pipeline failure should trigger heartbeat immediately

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #5727

          Activity

            People

              ibrusentsev Ivan Brusentsev
              ibrusentsev Ivan Brusentsev
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: