Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14349

Edit log may be rolled more frequently than necessary with multiple Standby nodes

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • ha, hdfs, qjm

    Description

      When HDFS-14317 was fixed, we tackled the problem that in a cluster with in-progress edit log tailing enabled, a Standby NameNode may never roll the edit logs, which can eventually cause data loss.

      Unfortunately, in the process, it was made so that if there are multiple Standby NameNodes, they will all roll the edit logs at their specified frequency, so the edit log will be rolled X times more frequently than they should be (where X is the number of Standby NNs). This is not as bad as the original bug since rolling frequently does not affect correctness or data availability, but may degrade performance by creating more edit log segments than necessary.

      Attachments

        Issue Links

          duplicates

          Sub-task - The sub-task of the issue HDFS-13189 Standby NameNode should roll active edit log when checkpointing

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is broken by

          Bug - A problem which impairs or prevents the functions of the product. HDFS-14317 Standby does not trigger edit log rolling when in-progress edit log tailing is enabled

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-14378 Simplify the design of multiple NN and both logic of edit log roll and checkpoint

          • Major - Major loss of function.
          • Patch Available

          Activity

            People

              ekanth Ekanth Sethuramalingam
              xkrogen Erik Krogen
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated: