Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6117

Hadoop ignores yarn.nodemanager.hostname for RPC listeners

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • 2.2.1, 2.4.1, 2.5.1
    • None
    • client, task
    • Any mapreduce example with standard cluster. In our case each node has four networks. It is important that all internode communication be done on a specific network.

    • Hide
      This patch has few new tests for the following reasons:
      TestTaskAttemptListenerImpl does not test or even perform the service start where the change is made. This is because that would starting a new process.
      TestMRClientService already checks the NM_HOST which change does effect.
      The change pulls the NM_HOST from the environment. This needs to be passed to a spawned process which none of the tests do.
      In general , it would be better if NM_HOST were more pervasive, that is, if the property were passed to the all the parts of the application, in particular the parts that deal with RPC. Since that is not the case, I have chosen to pull it from the environment where once can depend upon its being.

      I have tested it in clusters with multiple networks where the nm host is configured and those where it is not. It works as designed. That is, if the NM host is configured on the node the TaskAttempt Listner and the Client Service listen on the give NM host, otherwise they listen on the node's "hostname".
      Show
      This patch has few new tests for the following reasons: TestTaskAttemptListenerImpl does not test or even perform the service start where the change is made. This is because that would starting a new process. TestMRClientService already checks the NM_HOST which change does effect. The change pulls the NM_HOST from the environment. This needs to be passed to a spawned process which none of the tests do. In general , it would be better if NM_HOST were more pervasive, that is, if the property were passed to the all the parts of the application, in particular the parts that deal with RPC. Since that is not the case, I have chosen to pull it from the environment where once can depend upon its being. I have tested it in clusters with multiple networks where the nm host is configured and those where it is not. It works as designed. That is, if the NM host is configured on the node the TaskAttempt Listner and the Client Service listen on the give NM host, otherwise they listen on the node's "hostname".

    Description

      The RPC listeners for an application are using the hostname of the node as the binding address of the listener, They ignore yarn.nodemanager.hostname for this. In our setup we want all communication between nodes to be done via the network addresses we specify in yarn.nodemanager.hostname on each node.
      TaskAttemptListenerImpl.java and MRClientService.java are two places I have found where the default address is used rather that NM_host. The node Manager hostname should be used for all communication between nodes including the RPC listeners.

      Attachments

        1. MapReduce-534.patch
          5 kB
          Waldyn Benbenek
        2. MAPREDUCE-6117.002.patch
          5 kB
          Ray Chiang

        Activity

          People

            WaldynB Waldyn Benbenek
            WaldynB Waldyn Benbenek
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - 48h Original Estimate - 48h
                48h
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 384h
                384h