Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.2.0, 3.3.0
-
None
Description
When PYSPARK_DRIVER_PYTHON=$(which ipython) bin/pyspark is executed with Python >= 3.8, function registered wiht atexit seems to be executed in different scope than in Python 3.7.
It result in NameError: name 'sc' is not defined on exit:
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 3.3.0-SNAPSHOT
/_/
Using Python version 3.8.12 (default, Oct 12 2021 21:57:06)
Spark context Web UI available at http://192.168.0.198:4040
Spark context available as 'sc' (master = local[*], app id = local-1643555855409).
SparkSession available as 'spark'.
In [1]:
Do you really want to exit ([y]/n)? y
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/path/to/spark/python/pyspark/shell.py", line 49, in <lambda>
atexit.register(lambda: sc.stop())
NameError: name 'sc' is not defined
This could be easily fixed by capturing `sc` instance
diff --git a/python/pyspark/shell.py b/python/pyspark/shell.py index f0c487877a..4164e3ab0c 100644 --- a/python/pyspark/shell.py +++ b/python/pyspark/shell.py @@ -46,7 +46,7 @@ except Exception: sc = spark.sparkContext sql = spark.sql -atexit.register(lambda: sc.stop()) +atexit.register((lambda sc: lambda: sc.stop())(sc)) # for compatibility sqlContext = spark._wrapped
