Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7186

JobSubmitter#checkSpecs doesn not handle the situation when reduceNums is set less than zero.

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Patch Available
    • Minor
    • Resolution: Unresolved
    • None
    • None
    • client
    • None

    Description

      JobSubmitter#checkSpecs doesn not judge whether reduceNums is less than zero.

      private void checkSpecs(Job job) throws ClassNotFoundException,
      	InterruptedException, IOException {
      	JobConf jConf = (JobConf)job.getConfiguration();
      	// Check the output specification
      	if (jConf.getNumReduceTasks() == 0 ?
      	jConf.getUseNewMapper() : jConf.getUseNewReducer()) {
      	org.apache.hadoop.mapreduce.OutputFormat<?, ?> output =
      	ReflectionUtils.newInstance(job.getOutputFormatClass(),
      	job.getConfiguration());
      	output.checkOutputSpecs(job);
      	} else {
      	jConf.getOutputFormat().checkOutputSpecs(jtFs, jConf);
      	}
      	}
      	

      The code does handle the situation when reduceTaskNums is set to -1 or any other value less than zero.

      When use hive, the "mapred.reduce.tasks" has a default value -1.

       

      Attachments

        1. MAPREDUCE-7186-001.patch
          1 kB
          Hongyuan Li
        2. MAPREDUCE-7186-002.patch
          1 kB
          Hongyuan Li
        3. screenshot-1.png
          96 kB
          Hongyuan Li

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Hongyuan Li Hongyuan Li
            Hongyuan Li Hongyuan Li

            Dates

              Created:
              Updated:

              Slack

                Issue deployment