Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-11293

ExternalSorter and ExternalAppendOnlyMap should free shuffle memory in their stop() methods

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 1.3.1, 1.4.1, 1.5.1, 1.6.0
    • 1.6.0
    • Spark Core
    • None

    Description

      I discovered multiple leaks of shuffle memory while working on my memory manager consolidation patch, which added the ability to do strict memory leak detection for the bookkeeping that used to be performed by the ShuffleMemoryManager. This uncovered a handful of places where tasks can acquire execution/shuffle memory but never release it, starving themselves of memory.

      Problems that I found:

      • ExternalSorter.stop() should release the sorter's shuffle/execution memory.
      • BlockStoreShuffleReader should call ExternalSorter.stop() using a CompletionIterator.
      • ExternalAppendOnlyMap exposes no equivalent of stop() for freeing its resources.

      Attachments

        Issue Links

          Activity

            People

              joshrosen Josh Rosen
              joshrosen Josh Rosen
              Votes:
              14 Vote for this issue
              Watchers:
              36 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: