Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7224

Could not find or load main class org.apache.hadoop.yarn.server.nodemanager.containermanager.loca lizer.ContainerLocalizer when yarn on kerberos

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.0, 3.2.0
    • None
    • job submission
    • The environment:

      CentOS Linux release 7.5.1804 (Core) ;

      java version "1.8.0_171" ;

      Hadoop 3.1.0 ,3.2.0 ;

      kerberos 1.15.1-34 ;

       

    Description

       

      cat /etc/container-executor.cfg

      yarn.nodemanager.linux-container-executor.group=hadoop#
      banned.users=#comma separated list of users who can not run applications
      min.user.id=1#Prevent other super-users
      allowed.system.users=jingwei.shi#

       

      cat  yarn-site.xml

      <property>
      <name>yarn.resourcemanager.keytab</name>
      <value>/var/kerberos/krb5/user/hadoop.keytab</value>
      </property>

      <property>
      <name>yarn.resourcemanager.principal</name>
      <value>hadoop/_HOST@HADOOP.COM</value>
      </property>

      <property>
      <name>yarn.nodemanager.keytab</name>
      <value>/var/kerberos/krb5/user/hadoop.keytab</value>
      </property>

      <property>
      <name>yarn.nodemanager.principal</name>
      <value>hadoop/_HOST@HADOOP.COM</value>
      </property>

      <property>
      <name>yarn.nodemanager.default-container-executor.log-dirs.permissions</name>
      <value>777</value>
      </property>

      <property>
      <name>yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users</name>
      <value>false</value>
      </property>

      <property>
      <name>yarn.nodemanager.container-executor.class</name>
      <value>org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor</value>
      <!-value>org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor</value->
      </property>

      <property>
      <name>yarn.nodemanager.linux-container-executor.group</name>
      <value>hadoop</value>
      </property>

      <property>
      <name>yarn.nodemanager.local-dirs</name>
      <value>/tmp/userlogs</value>
      </property>

      <property>
      <name>yarn.nodemanager.log-dirs</name>
      <value>/tmp/userlogs</value>
      </property>

      ++++++++++++++++++++++++++++++++++

      klist
      Ticket cache: FILE:/tmp/krb5cc_1000
      Default principal: hadoop/bj-jd-dc-namenode-prod-0003.tendcloud.com@HADOOP.COM

      Valid starting Expires Service principal
      07/12/2019 16:05:00 07/13/2019 16:05:00 krbtgt/HADOOP.COM@HADOOP.COM
      renew until 07/26/2019 16:05:00

      Hadoop user submitted tasks successfully ;

      ————————————————————————————————

      while other user submitted tasks showed that no classes were found .

      klist
      Ticket cache: FILE:/tmp/krb5cc_1000
      Default principal: jingwei.shi/datanode-prod-0011.tendcloud.com@HADOOP.COM

      Valid starting Expires Service principal
      07/12/2019 14:50:37 07/13/2019 14:50:24 krbtgt/HADOOP.COM@HADOOP.COM
      renew until 07/26/2019 14:50:24

      2019-07-12 15:28:01,225 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed for container_e28_1562916339074_0001_02_000001
      java.io.IOException: Application application_1562916339074_0001 initialization failed (exitCode=1) with output: main : command provided 0
      main : run as user is jingwei.shi
      main : requested yarn user is jingwei.shi

      at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.startLocalizer(LinuxContainerExecutor.java:411)
      at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1245)
      Caused by: org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException: ExitCodeException exitCode=1: Error: Could not find or load main class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer

      Attachments

        Activity

          People

            Unassigned Unassigned
            hrsjw1 jingwei.shi
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: