Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-291

[Umbrella] Dynamic resource configuration

    XMLWordPrintableJSON

Details

    Description

      The current Hadoop YARN resource management logic assumes per node resource is static during the lifetime of the NM process. Allowing run-time configuration on per node resource will give us finer granularity of resource elasticity. This allows Hadoop workloads to coexist with other workloads on the same hardware efficiently, whether or not the environment is virtualized. More background and design details can be found in attached proposal.

      Attachments

        1. Elastic Resources for YARN-v0.2.pdf
          512 kB
          Junping Du
        2. YARN-291-AddClientRMProtocolToSetNodeResource-03.patch
          25 kB
          Junping Du
        3. YARN-291-all-v1.patch
          66 kB
          Junping Du
        4. YARN-291-CoreAndAdmin.patch
          46 kB
          Junping Du
        5. YARN-291-core-HeartBeatAndScheduler-01.patch
          31 kB
          Junping Du
        6. YARN-291-JMXInterfaceOnNM-02.patch
          4 kB
          Junping Du
        7. YARN-291-OnlyUpdateWhenResourceChange-01-fix.patch
          6 kB
          Junping Du
        8. YARN-291-YARNClientCommandline-04.patch
          8 kB
          Junping Du

        Issue Links

          1.
          Dynamic node resource configuration: core scheduler changes Sub-task Closed Junping Du
          2.
          Add updateNodeResource in ResourceManagerAdministrationProtocol Sub-task Closed Junping Du
          3.
          Add Admin API for supporting node resource configuration in command line Sub-task Resolved Íñigo Goiri
          4.
          REST API support for node resource configuration Sub-task Resolved Íñigo Goiri
          5.
          JMX support for node resource configuration Sub-task Open Unassigned
          6.
          Keep NM resource updated through dynamic resource config for RM/NM restart Sub-task Resolved Junping Du
          7.
          In case of long running tasks, reduce node resource should balloon out resource quickly by calling preemption API and suspending running task. Sub-task Resolved Íñigo Goiri
          8.
          Dynamic resource configuration feature can be configured to enable or disable and persistent on setting or not Sub-task Open Unassigned
          9.
          Replace set resource change on RMNode/SchedulerNode directly with event notification. Sub-task Resolved Junping Du
          10.
          Document Dynamic Resource Configuration feature Sub-task Open Young Chen
          11.
          Add admin ACL check to AdminService#updateNodeResource() Sub-task Resolved Kenji Kikushima
          12.
          ResouceOption's overcommitTimeout should be respected during resource update on NM Sub-task Resolved Unassigned
          13.
          Dynamic NM Resources Configuration file should be simplified. Sub-task Open Junping Du
          14.
          yarn rmadmin -updateNodeResource doesn't work Sub-task Resolved Junping Du
          15.
          updateNodeResource CLI should not accept negative values for resource Sub-task Resolved Junping Du
          16.
          RM Admin CLI for refreshNodesResources throws NPE when nothing is configured Sub-task Resolved Naganarasimha G R
          17.
          NM side resource value should get updated if change applied in RM side Sub-task Resolved Junping Du
          18.
          ContainersMonitorImpl need be adjusted when NM resource changed. Sub-task Resolved Íñigo Goiri

          Activity

            People

              junping_du Junping Du
              junping_du Junping Du
              Votes:
              10 Vote for this issue
              Watchers:
              72 Start watching this issue

              Dates

                Created:
                Updated: