Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-11422

fs2cs failes to generate valid capacity-scheduler.xml

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.4.0
    • None
    • capacityscheduler
    • None

    Description

      Attachments:

      Save them to /tmp/fair-scheduler.xml and /tmp/yarn-site.xml.

      Run the following command:

      rm -rf /tmp/output/*; yarn fs2cs --cluster-resource memory-mb=36201037824,vcores=4914 --no-terminal-rule-check -y /tmp/yarn-site.xml -f /tmp/fair-scheduler.xml -o  /tmp/output
      

      The conversion fails with the following error:

      2023-01-20 13:37:42,250 INFO service.AbstractService: Service org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler failed in state STOPPED
      java.lang.NullPointerException
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceStop(CapacityScheduler.java:428)
      	at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
      	at org.apache.hadoop.service.AbstractService.close(AbstractService.java:248)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.ConvertedConfigValidator.validateConvertedConfig(ConvertedConfigValidator.java:74)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:179)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverterMain.main(FSConfigToCSConfigConverterMain.java:47)
      2023-01-20 13:37:42,251 ERROR converter.ConvertedConfigValidator: Could not start Capacity Scheduler
      org.apache.hadoop.yarn.exceptions.YarnException: Failed to initialize queues
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:807)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:318)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:412)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.ConvertedConfigValidator.validateConvertedConfig(ConvertedConfigValidator.java:70)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:179)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverterMain.main(FSConfigToCSConfigConverterMain.java:47)
      	Suppressed: java.lang.NullPointerException
      		at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceStop(CapacityScheduler.java:428)
      		at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
      		at org.apache.hadoop.service.AbstractService.close(AbstractService.java:248)
      		at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.ConvertedConfigValidator.validateConvertedConfig(ConvertedConfigValidator.java:74)
      		... 2 more
      Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct queue ordering policy=fair queue=root.default
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:2053)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:172)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.<init>(ParentQueue.java:140)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.<init>(ParentQueue.java:115)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:276)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:283)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:176)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:798)
      	... 5 more
      Caused by: java.lang.ClassNotFoundException: fair
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
      	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
      	at java.lang.Class.forName0(Native Method)
      	at java.lang.Class.forName(Class.java:264)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:2049)
      	... 12 more
      2023-01-20 13:37:42,251 ERROR converter.FSConfigToCSConfigArgumentHandler: Verification failed: Failed to initialize queues
      Verification failed: Failed to initialize queues
      2023-01-20 13:37:42,251 ERROR converter.FSConfigToCSConfigConverterMain: Error while starting FS configuration conversion, see previous error messages for details!
      

      With the following manual changes on the generated capacity-scheduler.xml:

      • add: yarn.scheduler.capacity.root.users.auto-queue-creation-v2.enabled=true
      • remove: yarn.scheduler.capacity.root.default.auto-queue-creation-v2.enabled=true
      • remove: yarn.scheduler.capacity.root.users.ordering-policy=fair

      RM can start up.

      Attachments

        1. fair-scheduler-issue.xml
          4 kB
          Tamas Domok
        2. yarn-site.xml
          0.1 kB
          Tamas Domok

        Activity

          People

            tdomok Tamas Domok
            tdomok Tamas Domok
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: