Reuse SparkSession in pyspark via Gateway

In PySpark, we create SparkContext via user spark configurations or the default ones, and it gets launched through py4j gateway internally.

Let's say if I have launched py4j gateway from another application then to communicate with the same py4j gateway I have to set below configuration:-

export PYSPARK_GATEWAY_PORT=12345
export PYSPARK_GATEWAY_SECRET=***********************

So when PySpark tries to create its own SparkContext after the communication has been set up, it doesn't check whether there is any available SparkContext in the same JVM.

Current code snippet:-

def _initialize_context(self, jconf):
    """
    Initialize SparkContext in function to allow subclass specific initialization
    """
    return self._jvm.JavaSparkContext(jconf)

After changing it to the following, it works fine for me.

def _initialize_context(self, jconf):
    """
    Initialize SparkContext in function to allow subclass specific initialization
    """
    return self._jvm.JavaSparkContext(self._jvm.org.apache.spark.SparkContext.getOrCreate(jconf))

It looks like a good use case for improvement.