Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-1769

CapacityScheduler: Improve reservations

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.3.0
    • 2.6.0
    • capacityscheduler
    • None
    • Reviewed

    Description

      Currently the CapacityScheduler uses reservations in order to handle requests for large containers and the fact there might not currently be enough space available on a single host.

      The current algorithm for reservations is to reserve as many containers as currently required and then it will start to reserve more above that after a certain number of re-reservations (currently biased against larger containers). Anytime it hits the limit of number reserved it stops looking at any other nodes. This results in potentially missing nodes that have enough space to fullfill the request.

      The other place for improvement is currently reservations count against your queue capacity. If you have reservations you could hit the various limits which would then stop you from looking further at that node.

      The above 2 cases can cause an application requesting a larger container to take a long time to gets it resources.

      We could improve upon both of those by simply continuing to look at incoming nodes to see if we could potentially swap out a reservation for an actual allocation.

      Attachments

        1. YARN-1769.patch
          161 kB
          Thomas Graves
        2. YARN-1769.patch
          161 kB
          Thomas Graves
        3. YARN-1769.patch
          161 kB
          Thomas Graves
        4. YARN-1769.patch
          109 kB
          Thomas Graves
        5. YARN-1769.patch
          161 kB
          Thomas Graves
        6. YARN-1769.patch
          161 kB
          Chen He
        7. YARN-1769.patch
          152 kB
          Thomas Graves
        8. YARN-1769.patch
          152 kB
          Thomas Graves
        9. YARN-1769.patch
          152 kB
          Thomas Graves
        10. YARN-1769.patch
          151 kB
          Thomas Graves
        11. YARN-1769.patch
          148 kB
          Thomas Graves
        12. YARN-1769.patch
          152 kB
          Thomas Graves
        13. YARN-1769.patch
          153 kB
          Thomas Graves
        14. YARN-1769.patch
          153 kB
          Thomas Graves
        15. YARN-1769.patch
          153 kB
          Thomas Graves
        16. YARN-1769.patch
          153 kB
          Thomas Graves
        17. YARN-1769.patch
          153 kB
          Thomas Graves
        18. YARN-1769.patch
          153 kB
          Thomas Graves
        19. YARN-1769.patch
          152 kB
          Thomas Graves
        20. YARN-1769.patch
          152 kB
          Thomas Graves

        Activity

          People

            tgraves Thomas Graves
            tgraves Thomas Graves
            Votes:
            0 Vote for this issue
            Watchers:
            18 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: