Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7132

JobSplitWriter prints unnecessary warnings if EC(RS10,4) is used

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.1.1
    • Fix Version/s: 3.2.0, 3.0.4, 3.1.2
    • Component/s: client, mrv2
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently, JobSplitWriter compares the number of hosts for a certain block against a static value that comes from mapreduce.job.max.split.locations. The default value of this property is 10.

      However, an EC schema like RS-10-4 requires at least 14 hosts. In this case, 14 block locations will be returned and JobSplitWriter prints a warning, which can confuse users.

      A possible solution could check whether EC is enabled for a block and increase this value dynamically if needed. A simpler one is to simply increase the default value to a sensible number like 15.

       

        Attachments

        1. MAPREDUCE-7132-008.patch
          7 kB
          Peter Bacsko
        2. MAPREDUCE-7132-007.patch
          6 kB
          Peter Bacsko
        3. MAPREDUCE-7132-006.patch
          6 kB
          Peter Bacsko
        4. MAPREDUCE-7132-005.patch
          12 kB
          Peter Bacsko
        5. MAPREDUCE-7132-004.patch
          12 kB
          Peter Bacsko
        6. MAPREDUCE-7132-003.patch
          12 kB
          Peter Bacsko
        7. MAPREDUCE-7132-002.patch
          12 kB
          Peter Bacsko
        8. MAPREDUCE-7132-001.patch
          12 kB
          Peter Bacsko

          Activity

            People

            • Assignee:
              pbacsko Peter Bacsko
              Reporter:
              pbacsko Peter Bacsko
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: