[SPARK-27434] memory leak in spark driver - ASF JIRA

XML

Word

Printable

JSON

Type: Bug
Status: Resolved
Priority: Major
Resolution: Cannot Reproduce
Affects Version/s: 2.4.0
Fix Version/s: None
Component/s: Spark Core
Labels:
None
Environment:

Hide

OS: Centos 7

JVM:

**openjdk version "1.8.0_201"
OpenJDK Runtime Environment (IcedTea 3.11.0) (Alpine 8.201.08-r0)
OpenJDK 64-Bit Server VM (build 25.201-b08, mixed mode)

Spark version: 2.4.0

Show
OS: Centos 7 JVM: ** openjdk version "1.8.0_201" OpenJDK Runtime Environment (IcedTea 3.11.0) (Alpine 8.201.08-r0) OpenJDK 64-Bit Server VM (build 25.201-b08, mixed mode) Spark version: 2.4.0

we got a OOM exception on the driver after driver has completed multiple jobs(we are reusing spark context).

so we took a heap dump and looked at the leak analysis, found out that under AsyncEventQueue there are 3.5GB of heap allocated. Possibly a leak.

can someone take a look at?

here is the heap analysis:

links to

GitHub Pull Request #24461