Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-11643

Skip unnecessary pre-check in Multi Node Placement

    XMLWordPrintableJSON

Details

    Description

      When Multi Node Placement enabled, RegularContainerAllocator do a while loop to find one node from candidate set to allocate for a given scheduler key. Before do allocate, pre-check be called to check if current node satisfies check. If this node does not pass all checks, just continue to next node.

      if (reservedContainer == null) {
        result = preCheckForNodeCandidateSet(node,
            schedulingMode, resourceLimits, schedulerKey);
        if (null != result) {
          continue;
        }
      } 

      But some checks are related to scheduler Key or Application which return PRIORITY_SKIPPED or APP_SKIPPED. It means that if first node does not pass check, the following nodes also do not pass.
      If cluster have 5000 nodes in default partition, Scheduler will waste 5000 times loop for just one scheduler key.

       

      Attachments

        Issue Links

          Activity

            People

              yifan.stan Xie YiFan
              yifan.stan Xie YiFan
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: