Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7132

JobSplitWriter prints unnecessary warnings if EC(RS10,4) is used

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.1
    • 3.2.0, 3.0.4, 3.1.2
    • client, mrv2
    • None
    • Reviewed

    Description

      Currently, JobSplitWriter compares the number of hosts for a certain block against a static value that comes from mapreduce.job.max.split.locations. The default value of this property is 10.

      However, an EC schema like RS-10-4 requires at least 14 hosts. In this case, 14 block locations will be returned and JobSplitWriter prints a warning, which can confuse users.

      A possible solution could check whether EC is enabled for a block and increase this value dynamically if needed. A simpler one is to simply increase the default value to a sensible number like 15.

       

      Attachments

        1. MAPREDUCE-7132-001.patch
          12 kB
          Peter Bacsko
        2. MAPREDUCE-7132-002.patch
          12 kB
          Peter Bacsko
        3. MAPREDUCE-7132-003.patch
          12 kB
          Peter Bacsko
        4. MAPREDUCE-7132-004.patch
          12 kB
          Peter Bacsko
        5. MAPREDUCE-7132-005.patch
          12 kB
          Peter Bacsko
        6. MAPREDUCE-7132-006.patch
          6 kB
          Peter Bacsko
        7. MAPREDUCE-7132-007.patch
          6 kB
          Peter Bacsko
        8. MAPREDUCE-7132-008.patch
          7 kB
          Peter Bacsko

        Activity

          People

            pbacsko Peter Bacsko
            pbacsko Peter Bacsko
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: