Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-17237

Remove IPCLoggerChannel Metrics when the logger is closed

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      When an IPCLoggerChannel is created (which is used to read from and write to the Journal nodes) it also creates a metrics object. When the namenodes failover, the IPC loggers are all closed and reopened in read mode on the new SBNN or the read mode is closed on the SBNN and re-opened in write mode. The closing frees the resources and discards the original IPCLoggerChannel object and causes a new one to be created by the caller.

      If a Journal node was down and added back to the cluster with the same hostname, but a different IP, when the failover happens, you end up with 4 metrics objects for the JNs:

      1. For for each of the original 3 IPs
      2. One for the new IP

      The old stale metric will remain forever and will no longer be updated, leading to confusing results in any tools that use the metrics for monitoring.

      This change, ensures we un-register the metrics when the logger channel is closed and a new metrics object gets created when the new channel is created.

      I have added a small test to prove this, but also reproduced the original issue on a docker cluster and validated it is resolved with this change in place.

      For info, the logger metrics look like:

      {
         "name" : "Hadoop:service=NameNode,name=IPCLoggerChannel-192.168.32.8-8485",
          "modelerType" : "IPCLoggerChannel-192.168.32.8-8485",
          "tag.Context" : "dfs",
          "tag.IsOutOfSync" : "false",
          "tag.Hostname" : "957e3e66f10b",
          "QueuedEditsSize" : 0,
          "LagTimeMillis" : 0,
          "CurrentLagTxns" : 0
        }
      

      Node the name includes the IP, rather than the hostname.

      Attachments

        Issue Links

          Activity

            People

              sodonnell Stephen O'Donnell
              sodonnell Stephen O'Donnell
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: