Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Implemented
-
3.5.0
-
None
Description
Step1. start a spark connect server
Step2. submit a spark job which will run long
from pyspark.sql import SparkSession spark = SparkSession.builder.remote(f"sc://HOST:PORT/;user_id=job").create() spark.sql("A SQL will run longer than creating 100 sessions").show()
Step3. create more than 100 sessions
Tips: Run concurrently with step2
for i in range(0, 200): spark = SparkSession.builder.remote(f"sc://HOST:PORT/;user_id={i}").create() spark.sql("show databases")
When the python code in step3 is executed, the session created in step2 will be evicted, and the client will wait forever
Attachments
Issue Links
- links to