Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-386

[Umbrella] YARN API Changes

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      This is the umbrella ticket to capture any and every API cleanup that we wish to do before YARN can be deemed beta/stable. Doing this API cleanup now and ASAP will help us escape the pain of supporting bad APIs in beta/stable releases.

      Attachments

        Issue Links

          is blocked by

          Sub-task - The sub-task of the issue YARN-85 Allow per job log aggregation configuration

          • Critical - Crashes, loss of data, severe memory leak.
          • Open

          Sub-task - The sub-task of the issue YARN-352 Inconsistent picture of how a container was killed when querying RM and NM in case of preemption

          • Major - Major loss of function.
          • Open

          Sub-task - The sub-task of the issue YARN-221 NM should provide a way for AM to tell it not to aggregate logs.

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue YARN-1279 Expose a client API to allow clients to figure if log aggregation is complete

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue YARN-475 Remove ApplicationConstants.AM_APP_ATTEMPT_ID_ENV as it is no longer set in an AM's environment

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. YARN-550 ContainerLaunchContext::getServiceData and setServiceData need a lot of clarification

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. YARN-662 [Umbrella] Enforce required parameters for all the protocols

          • Major - Major loss of function.
          • Open
          relates to

          Sub-task - The sub-task of the issue YARN-435 Make it easier to access cluster topology information in an AM

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. YARN-723 Yarn default value of physical cpu cores to virtual core is 2

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. YARN-142 [Umbrella] Cleanup YARN APIs w.r.t exceptions

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed
          1.
          		 Fix inconsistent protocol naming Sub-task Closed Vinod Kumar Vavilapalli
          2.
          		 Rationalize AllocateResponse in RM scheduler API Sub-task Closed Zhijie Shen
          3.
          		 ApplicationMaster retry times should be set by Client Sub-task Closed Zhijie Shen
          4.
          		 Improve documentation about what "asks" means in AMRMProtocol Sub-task Open Sandy Ryza
          5.
          		 RM should throw different exceptions for while querying app/node/queue Sub-task Open Vinod Kumar Vavilapalli
          6.
          		 Flatten NodeHeartbeatResponse Sub-task Closed Xuan Gong
          7.
          		 Flatten RegisterNodeManagerResponse Sub-task Closed Xuan Gong
          8.
          		 Clean up unused collection methods in various APIs Sub-task Closed Xuan Gong
          9.
          		 The ID classes should be immutable Sub-task Resolved Unassigned
          10.
          		 Move special container exit codes from YarnConfiguration to API Sub-task Closed Sandy Ryza
          11.
          		 Define value for * in the scheduling protocol Sub-task Closed Zhijie Shen
          12.
          		 Change startContainer NM API to accept Container as a parameter and make ContainerLaunchContext user land Sub-task Closed Xuan Gong
          13.
          		 Remove ApplicationConstants.AM_APP_ATTEMPT_ID_ENV as it is no longer set in an AM's environment Sub-task Closed Hitesh Shah
          14.
          		 Make it easier to access cluster topology information in an AM Sub-task Open Omkar Vinit Joshi
          15.
          		 Make IDs read only Sub-task Resolved Siddharth Seth
          16.
          		 Remove ContainerStatus, ContainerState from Container api interface as they will not be called by the container object Sub-task Closed Xuan Gong
          17.
          		 AggregatedLogFormat should be marked Private / Unstable Sub-task Resolved Siddharth Seth
          18.
          		 YarnClient.submitApplication should wait for application to be accepted by the RM Sub-task Closed Zhijie Shen
          19.
          		 Expose resource metrics as part of YarnClusterMetrics Sub-task Open Omkar Vinit Joshi
          20.
          		 Have YarnClient generate a directly usable ApplicationSubmissionContext Sub-task Closed Karthik Kambatla
          21.
          		 Make all YARN API and libraries available through an api jar Sub-task Open Vinod Kumar Vavilapalli
          22.
          		 Nodemanager should set some key information into the environment of every container that it launches. Sub-task Closed Xuan Gong
          23.
          		 Setter of xxxxPBImpl doesn't really set the collection, but append its items to an existing collection Sub-task Resolved Zhijie Shen
          24.
          		 Add application type to ApplicationReport Sub-task Closed Mayank Bansal
          25.
          		 User should not be part of ContainerLaunchContext Sub-task Closed Omkar Vinit Joshi
          26.
          		 Remove duplication of data in Container Sub-task Resolved Zhijie Shen
          27.
          		 ApplicationReport does not provide progress value of application Sub-task Closed Hitesh Shah
          28.
          		 RM recovery related records do not belong to the API Sub-task Closed Vinod Kumar Vavilapalli
          29.
          		 Cleanup BuilderUtils Sub-task Open Unassigned
          30.
          		 ContainerLaunchContext.containerTokens should simply be called tokens Sub-task Closed Vinod Kumar Vavilapalli
          31.
          		 Add application type to submission context for map reduce Sub-task Resolved Mayank Bansal
          32.
          		 ContainerManager.startContainer needs to only have ContainerTokenIdentifier instead of the whole Container Sub-task Closed Vinod Kumar Vavilapalli
          33.
          		 Flatten NodeReport Sub-task Closed Sandy Ryza
          34.
          		 masterContainer and status are in ApplicationReportProto but not in ApplicationReport Sub-task Closed Zhijie Shen
          35.
          		 Copy BuilderUtil methods into individual records Sub-task Closed Jian He
          36.
          		 AMRM protocol changes for sending NMToken list Sub-task Closed Omkar Vinit Joshi
          37.
          		 Copy BuilderUtil methods into token-related records Sub-task Closed Jian He
          38.
          		 Move RMIdentifier from Container to ContainerTokenIdentifier Sub-task Closed Vinod Kumar Vavilapalli
          39.
          		 container-log4j.properties should not refer to mapreduce properties Sub-task Closed Zhijie Shen
          40.
          		 AMRMClient should have a separate setProgress instead of sending progress as part of allocate Sub-task Open Vinod Kumar Vavilapalli
          41.
          		 Rename ResourceRequest (get,set)HostName to (get,set)ResourceName Sub-task Closed Arun Murthy
          42.
          		 Rename AllocateResponse.reboot to AllocateResponse.resync Sub-task Closed Bikas Saha
          43.
          		 Create Command enum in AllocateResponse Sub-task Closed Bikas Saha
          44.
          		 Move PreemptionContainer/PremptionContract/PreemptionMessage/StrictPreemptionContract/PreemptionResourceRequest to api.records Sub-task Closed Jian He
          45.
          		 Add individual factory method for api protocol records Sub-task Closed Jian He
          46.
          		 Move BuilderUtils from yarn-common to yarn-server-common Sub-task Closed Jian He
          47.
          		 Make ApplicationAttemptID, ContainerID, NodeID immutable Sub-task Closed Jian He
          48.
          		 Some Exceptions no longer need to be wrapped by YarnException and can be directly thrown out after YARN-142 Sub-task Closed Jian He
          49.
          		 Move ProtoBase from api.records to api.records.impl.pb Sub-task Closed Jian He
          50.
          		 Move YarnRuntimeException from package api.yarn to api.yarn.exceptions Sub-task Closed Jian He
          51.
          		 Investigate adding a more generic error field to AMResponse Sub-task Open Unassigned
          52.
          		 ClientRMProtocol.getAllApplications should accept ApplicationType as a parameter Sub-task Closed Xuan Gong
          53.
          		 Remove unreferenced objects from proto Sub-task Closed Jian He
          54.
          		 Expose LOGDIR that containers should use for logging Sub-task Closed Jian He
          55.
          		 Ensure that RM RPC APIs that return nodes are consistent with /nodes REST API Sub-task Closed Sandy Ryza
          56.
          		 Move NodeHealthStatus from yarn.api.record to yarn.server.api.record Sub-task Closed Jian He
          57.
          		 AMRMClient support for resource blacklisting Sub-task Closed Junping Du
          58.
          		 Augment AM - RM client module to be able to request containers only at specific locations Sub-task Closed Sandy Ryza
          59.
          		 Remove resource min from Yarn client API Sub-task Closed Alejandro Abdelnur
          60.
          		 Fix yarn-api javadoc annotations Sub-task Closed Jian He
          61.
          		 Move ContainerExitStatus from yarn.api to yarn.api.records Sub-task Closed Jian He
          62.
          		 YARN application classpath should add $PWD/* in addition to $PWD Sub-task Open Jian He
          63.
          		 Rename FinishApplicationMasterRequest.setFinishApplicationStatus to setFinalApplicationStatus to be consistent with getter Sub-task Closed Jian He
          64.
          		 Move RMAdmin from yarn.client to yarn.client.cli and rename as RMAdminCLI Sub-task Closed Jian He
          65.
          		 Add static factory to yarn client lib interface and change it to abstract class Sub-task Closed Jian He
          66.
          		 Fix yarn-common javadoc annotations Sub-task Closed Vinod Kumar Vavilapalli
          67.
          		 Need to make Resource arithmetic methods accessible Sub-task Closed Jian He
          68.
          		 Remove resource min from GetNewApplicationResponse Sub-task Closed Jian He
          69.
          		 Update Resource javadoc to clarify units for memory Sub-task Resolved Unassigned
          70.
          		 Review/fix annotations for yarn-client module and clearly differentiate *Async apis Sub-task Closed Zhijie Shen
          71.
          		 Move Clock/SystemClock to util package Sub-task Closed Zhijie Shen
          72.
          		 The name 'ApplicationTokenIdentifier' is misleading Sub-task Resolved Vinod Kumar Vavilapalli
          73.
          		 Rename ApplicationToken to AMRMToken Sub-task Closed Omkar Vinit Joshi
          74.
          		 Move ProtoUtils to yarn.api.records.pb.impl Sub-task Closed Jian He
          75.
          		 ClusterInfo.java doesn't seem to belong to org.apache.hadoop.yarn Sub-task Closed Zhijie Shen
          76.
          		 Annotate and document AuxService APIs Sub-task Closed Vinod Kumar Vavilapalli
          77.
          		 Move pb Impl from yarn-api to yarn-common Sub-task Closed Jian He
          78.
          		 Rename getClusterAvailableResources to getAvailableResources in AMRMClients Sub-task Closed Jian He
          79.
          		 Move Resources and the calculators to a new package util.resource in yarn-common Sub-task Resolved Jian He
          80.
          		 YARNClient.getApplicationReport(unknownAppId) returns a null report Sub-task Closed Xuan Gong
          81.
          		 Consider adding clone methods to protocol objects Sub-task Open Junping Du
          82.
          		 yarn.application.classpath requires client to know service internals Sub-task Open Sandy Ryza
          83.
          		 ApplicationReport can return tokens only when users asks for it explicitly Sub-task Open Unassigned
          84.
          		 Deprecate/ mark private few methods from YarnConfiguration Sub-task Resolved Unassigned
          85.
          		 ApplicationContants should be a proto definition instead of being a java class Sub-task Open Unassigned
          86.
          		 FinishApplicationMasterRequest should also have a final IPC/RPC address. Sub-task Open Junping Du
          87.
          		 NM silently ignores non-existent service in StartContainerRequest Sub-task Closed Xuan Gong
          88.
          		 AMRMClientAsync missing blacklist addition and removal functionality Sub-task Closed Bartosz Ługowski

          Activity

            People

              vinodkv Vinod Kumar Vavilapalli
              vinodkv Vinod Kumar Vavilapalli
              Votes:
              0 Vote for this issue
              Watchers:
              24 Start watching this issue

              Dates

                Created:
                Updated: