Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7457

Limit number of spill files getting created

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 3.4.0
    • 3.4.0
    • mrv2
    • Reviewed

    Description

      Hi,

       

      We have been facing some issues where many of our cluster node disks go full because of some rogue applications creating a lot of spill data

      We wanted to fail the app if more than a threshold amount of spill files are written

      Please let us know if any such capability is supported

       

      If the capability is not there, we are proposing it to support it via a config, we have added a PR for the same: https://github.com/apache/hadoop/pull/6155  please let us know your thoughts on it

      Attachments

        Issue Links

          Activity

            People

              mudit-97 Mudit Sharma
              mudit-97 Mudit Sharma
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: