Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-2812

Spill InternalCachedBag into only 1 file

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • data
    • None
    • Patch Available

    Description

      I encountered a reducer's OOM because of java.io.DeleteOnExitHook. And I found out that the InternalCachedBag creates a seperate tmp file, and the tmp files is deleted on exit. So the file delete hook caused the OOM.
      Why not just hold the tmp file handle and spill only one tmp file?
      Too many tmp files may block the tasktracker start process, if the tmp files are not cleaned on time and the tasktracker restarts at this specific time.

      Attachments

        1. aa.jpg
          102 kB
          Haitao Yao
        2. spill.patch
          2 kB
          Haitao Yao

        Activity

          People

            haitao.yao Haitao Yao
            haitao.yao Haitao Yao
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated: