Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Invalid
-
2.4.0
-
None
-
None
Description
Machine OS: Ubuntu 16.04.
Kubernetes: Minikube
Kubernetes Version: 1.10.0
Spark Kubernetes Image: pyspark ( at docker hub: saivarunr/spark-py:2.4 ) built using standard spark docker build.sh file.
Driver is inside pod in kubernetes cluster.
Steps to replicate:
1) Create a spark Session:
// spark_session=SparkSession.builder.master('k8s://https://192.168.99.100:8443').config('spark.executor.instances','1').config('spark.kubernetes.container.image','saivarunr/spark-py:2.4').getOrCreate()
2) Create a sample DataFrame
// df=spark_session.createDataFrame([{'a':1}])
3) Do some operation on this dataframe
// df.count()
I get this output.
// Exception happened during processing of request from ('127.0.0.1', 38690) Traceback (most recent call last): File "/usr/lib/python3.6/socketserver.py", line 317, in _handle_request_noblock self.process_request(request, client_address) File "/usr/lib/python3.6/socketserver.py", line 348, in process_request self.finish_request(request, client_address) File "/usr/lib/python3.6/socketserver.py", line 361, in finish_request self.RequestHandlerClass(request, client_address, self) File "/usr/lib/python3.6/socketserver.py", line 721, in __init__ self.handle() File "/spark-2.4.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/accumulators.py", line 266, in handle poll(authenticate_and_accum_updates) File "/spark-2.4.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/accumulators.py", line 241, in poll if func(): File "/spark-2.4.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/accumulators.py", line 254, in authenticate_and_accum_updates received_token = self.rfile.read(len(auth_token)) TypeError: object of type 'NoneType' has no len()
4) Repeat above step; it won't show the error.
But now close the session, kill the python terminal or process. and try again, the same happens.
Something related to https://issues.apache.org/jira/browse/SPARK-26019 ?
Attachments
Issue Links
- is duplicated by
-
-
- Resolved
-