Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4256

Improve resource scheduling

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • None
    • None
    • resourcemanager
    • None

    Description

      Currently resource manager supports only Memory resource during container allocation.

      I propose following improvements:

      1. add support for CPU utilization. Node CPU used information can be obtained by ResourceCalculatorPlugin.

      2. add support for custom resources. In node configuration will be something like:

      name=node.resource.GPU, value=1 (node has 1 GPU).

      If job will need to use GPU for computation, it will add "GPU=1" requirement to its job config and Resource Manager will allocate container on node with GPU available.

      Attachments

        Issue Links

          is blocked by

          New Feature - A new feature of the product, which has yet to be developed. YARN-2 Enhance CS to schedule accounting for both memory and cpu cores

          • Major - Major loss of function.
          • Closed
          relates to

          Sub-task - The sub-task of the issue YARN-4 Make ContainersMonitor pluggable

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue YARN-3 Add support for CPU isolation/monitoring of containers

          • Major - Major loss of function.
          • Closed

          Activity

            People

              Unassigned Unassigned
              hsn Radim Kolar
              Votes:
              0 Vote for this issue
              Watchers:
              27 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: