Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-29301

Removing block is not reflected to the driver/executor's storage memory

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 2.4.4, 3.0.0
    • None
    • Spark Core
    • None

    Description

      While investigating SPARK-29055 I've found that heap memory in driver doesn't increase which would denote memory leak happen, but storage memory doesn't decrease even there're some broadcast blocks getting removed - these values keep increasing.

      Please refer SPARK-29055 to see the step to reproduce. Very easy to see it, as running simple query repeatedly would reproduce the issue.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              kabhwan Jungtaek Lim
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: