Dynamic Queue ACL handling in Legacy and Flexible Auto Created Queues

During the investigation YARN-10922 I found multiple issues with dynamic ACL handling, no wonder it was documented as unsupported. But in some cases it did work and some users utilised that behaviour.

There is a severe bug when using Flexible AQC and dynamic parent queues, anybody can submit an app regardless of what ACLs are set (when the dynamic parent queue is not exists yet) - see: YARN-11066.

After this dynamic ACL feature YARN-11060 should be fixed as well.

Legacy AQC:

yarn.scheduler.capacity.root.managed.auto-create-child-queue.enabled: true

yarn.scheduler.capacity.root.managed.acl_submit_applications: nobody
yarn.scheduler.capacity.root.managed.acl_administer_queue: nobody

yarn.scheduler.capacity.root.managed.acl_submit_applications: user1
yarn.scheduler.capacity.root.managed.acl_administer_queue: admin1

yarn.scheduler.capacity.root.managed.leaf-queue-template.acl_submit_applications: user2
yarn.scheduler.capacity.root.managed.leaf-queue-template.acl_administer_queue: admin2

user2 can submit an application to root.managed.user2 even if the queue doesn't exist yet due to template

Permissions:

• root: {nobody, nobody}
• root.managed: {user1, admin1}
• root.managed.auto: {user2, admin2}

Flexible AQC:

yarn.scheduler.capacity.root.auto-queue-creation-v2.enabled: true

yarn.scheduler.capacity.root.acl_submit_applications: user
yarn.scheduler.capacity.root.acl_administer_queue: admin

yarn.scheduler.capacity.root.auto-queue-creation-v2.parent-template.acl_submit_applications: parentUser1
yarn.scheduler.capacity.root.auto-queue-creation-v2.parent-template.acl_administer_queue: parentAdmin1

yarn.scheduler.capacity.root.*.auto-queue-creation-v2.leaf-template.acl_submit_applications: user1
yarn.scheduler.capacity.root.*.auto-queue-creation-v2.leaf-template.acl_administer_queue: admin1

user1 can submit an application to root.autoParent.user1 even if the queue doesn't exist yet due to template

Permissions:

• root: {user, admin}
• root.autoParent: {parentUser1, parentAdmin1}
• root.autoParent.autoLeaf: {user1, admin1}

NOTE: the .leaf-template and .parent-template overwrites the .template variant.

The Scheduler Response should contain the correct values for the queueAcls field for the dynamic queues.

The wildcard pattern for flexible AQC was not working properly for root.* queue pattern.