Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-16792

Add -newEditsOnly option to name node initializeSharedEdits command to format unformatted journal nodes for master replacement

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.3.3, 3.3.4
    • None
    • journal-node

    Description

      Add -newEditsOnly option to name node initializeSharedEdits command to format unformatted journal nodes for master replacement.

       

      The current hadoop has limitations on formatting journal nodes. First, it requires all journal nodes to be present when formatting name node. After that, new journal nodes could not be formatted unless we reformat all of them and destroy all HDFS edit logs.

       

      $ hdfs namenode -initializeSharedEdits -newEditsOnly

      There are two cases here:

      1) the replaced master instance has name node running on it, then the above command will only be run once on the existing name node.

      2) the replaced master instance does not have a name node running on it, then the above command will be run twice. But the above command is idempotent as long as the same command will not be issued at the same time, which could be guaranteed by instance controller when reconfiguring master instances. The reason is that all hadoop name node commands such as format, bootstrap, and initializeSharedEdits are not designed to be thread safe and they rely on an external mechanism, which is a manual work for hadoop admin for on-premise clusters, to make sure that they are not executed concurrently. We still keep the same behavior of the initializeSharedEdits command to minimize the code/logic changes.

      Attachments

        Issue Links

          Activity

            People

              groot Ashutosh Gupta
              groot Ashutosh Gupta
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: