Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7410

Expose API to get task ids and individual task report given task Id from org.apache.hadoop.mapreduce.Job

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 2.10.2, 3.3.4
    • None
    • jobhistoryserver, yarn
    • None

    Description

      Currently org.apache.hadoop.mapreduce.Job exposes getTaskReports(TaskType) API to fetch task reports of either mapper or reducer. However for MR jobs with large number of tasks this causes OOM issues while fetching all task reports as seen with JHS (HistoryClientService.getTaskReports), HistoryClientService also exposes an API getTaskReport(TaskId) where a TaskId can be provided within the GetTaskReportRequest. org.apache.hadoop.mapreduce.Job can expose 2 API so that individual task report can be fetched after listing them from client side

      1. Job.getTasks(TaskType) -> List<TaskId> - This would return TaskId of all tasks with given Type to the client
      2. Job.getTaskReport(TaskId) -> TaskReport - This would return task report for single task to the client

      For JHS since JobHistoryParser.parse already parses full history file by default and maintains the list of tasks within JobHistoryParser.JobInfo's tasksMap, this info should be easy to get

      One additional thing that needs to be seen is if this can be supported for requests which are redirected to MRClientService (within MRAppMaster) for running jobs

      Attachments

        Activity

          People

            Unassigned Unassigned
            ukumar Ujjawal Kumar
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: