Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.4.0
-
None
-
None
Description
Attachments:
Save them to /tmp/fair-scheduler.xml and /tmp/yarn-site.xml.
Run the following command:
rm -rf /tmp/output/*; yarn fs2cs --cluster-resource memory-mb=36201037824,vcores=4914 --no-terminal-rule-check -y /tmp/yarn-site.xml -f /tmp/fair-scheduler.xml -o /tmp/output
The conversion fails with the following error:
2023-01-20 13:37:42,250 INFO service.AbstractService: Service org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler failed in state STOPPED java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceStop(CapacityScheduler.java:428) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221) at org.apache.hadoop.service.AbstractService.close(AbstractService.java:248) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.ConvertedConfigValidator.validateConvertedConfig(ConvertedConfigValidator.java:74) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:179) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverterMain.main(FSConfigToCSConfigConverterMain.java:47) 2023-01-20 13:37:42,251 ERROR converter.ConvertedConfigValidator: Could not start Capacity Scheduler org.apache.hadoop.yarn.exceptions.YarnException: Failed to initialize queues at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:807) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:318) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:412) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.ConvertedConfigValidator.validateConvertedConfig(ConvertedConfigValidator.java:70) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:179) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverterMain.main(FSConfigToCSConfigConverterMain.java:47) Suppressed: java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceStop(CapacityScheduler.java:428) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221) at org.apache.hadoop.service.AbstractService.close(AbstractService.java:248) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.ConvertedConfigValidator.validateConvertedConfig(ConvertedConfigValidator.java:74) ... 2 more Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct queue ordering policy=fair queue=root.default at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:2053) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:172) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.<init>(ParentQueue.java:140) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.<init>(ParentQueue.java:115) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:276) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:283) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:176) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:798) ... 5 more Caused by: java.lang.ClassNotFoundException: fair at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:2049) ... 12 more 2023-01-20 13:37:42,251 ERROR converter.FSConfigToCSConfigArgumentHandler: Verification failed: Failed to initialize queues Verification failed: Failed to initialize queues 2023-01-20 13:37:42,251 ERROR converter.FSConfigToCSConfigConverterMain: Error while starting FS configuration conversion, see previous error messages for details!
With the following manual changes on the generated capacity-scheduler.xml:
- add: yarn.scheduler.capacity.root.users.auto-queue-creation-v2.enabled=true
- remove: yarn.scheduler.capacity.root.default.auto-queue-creation-v2.enabled=true
- remove: yarn.scheduler.capacity.root.users.ordering-policy=fair
RM can start up.