Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12943 Consistent Reads from Standby Node
  3. HDFS-13150

[Edit Tail Fast Path] Allow SbNN to tail in-progress edits from JN via RPC

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      In the interest of making coordinated/consistent reads easier to complete with low latency, it is advantageous to reduce the time between when a transaction is applied on the ANN and when it is applied on the SbNN. We propose adding a new "fast path" which can be used to tail edits when low latency is desired. We leave the existing tailing logic in place, and fall back to this path on startup, recovery, and when the fast path encounters unrecoverable errors.

      Attachments

        1. edit-tailing-fast-path-design-v2.pdf
          172 kB
          Erik Krogen
        2. edit-tailing-fast-path-design-v1.pdf
          171 kB
          Erik Krogen
        3. edit-tailing-fast-path-design-v0.pdf
          165 kB
          Erik Krogen

        Issue Links

        depends upon

        Sub-task - The sub-task of the issue HDFS-13607 [Edit Tail Fast Path Pt 1] Enhance JournalNode with an in-memory cache of recent edit transactions

        • Major - Major loss of function.
        • Resolved

        Sub-task - The sub-task of the issue HDFS-13608 [Edit Tail Fast Path Pt 2] Add ability for JournalNode to serve edits via RPC

        • Major - Major loss of function.
        • Resolved

        Sub-task - The sub-task of the issue HDFS-13609 [Edit Tail Fast Path Pt 3] NameNode-side changes to support tailing edits via RPC

        • Major - Major loss of function.
        • Resolved

        Sub-task - The sub-task of the issue HDFS-13610 [Edit Tail Fast Path Pt 4] Cleanup: integration test, documentation, remove unnecessary dummy sync

        • Major - Major loss of function.
        • Resolved
        is related to

        Improvement - An improvement or enhancement to an existing feature or task. HADOOP-16208 Do Not Log InterruptedException in Client

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved
        relates to

        Sub-task - The sub-task of the issue HDFS-13789 Reduce logging frequency of QuorumJournalManager#selectInputStreams

        • Trivial - Cosmetic problem like misspelt words or misaligned text.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HDFS-16493 [SBN Read]When fast path tail enabled, standby or observer namenode may read uncommitted data

        • Critical - Crashes, loss of data, severe memory leak.
        • Open

        Bug - A problem which impairs or prevents the functions of the product. HDFS-14500 NameNode StartupProgress continues to report edit log segments after the LOADING_EDITS phase is finished

        • Major - Major loss of function.
        • Resolved

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            xkrogen Erik Krogen
            xkrogen Erik Krogen
            Votes:
            0 Vote for this issue
            Watchers:
            15 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment