Possible memory leak in Spark

This issue is a clone of the (SPARK-29055). After Spark version 2.3.3, I observe that the JVM memory is increasing slightly overtime. This behavior also affects the application performance because when I run my real application in testing environment, after a while the persisted dataframes stop fitting into the executors memory and I have spill to disk.

JVM memory usage (based on htop command)

HOW TO REPRODUCE THIS BEHAVIOR:

 Reproduce the above behavior, by running the snippet code (I prefer to run without any sleeping delay) and track the JVM memory with top or htop command.

import time
import os

from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql import types as T

target_dir = "..."

spark=SparkSession.builder.appName("DataframeCount").getOrCreate()

while True:
    for f in os.listdir(target_dir):
        df = spark.read.load(target_dir + f, format="csv")
        print("Number of records: {0}".format(df.count()))
        time.sleep(15)

TESTED CASES WITH THE SAME BEHAVIOUR:

• I tested with default settings (spark-defaults.conf)
• Add spark.cleaner.periodicGC.interval 1min (or less)
• Turn spark.cleaner.referenceTracking.blocking=false
• Run the application in cluster mode
• Increase/decrease the resources of the executors and driver
• I tested with extraJavaOptions in driver and executor -XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=12
• It is also tested with the Spark 2.4.4 (latest) and had the same behavior.
   

DEPENDENCIES

• Operation system: Ubuntu 16.04.3 LTS
• Java: jdk1.8.0_131 (tested also with jdk1.8.0_221)
• Python: Python 2.7.12