Description
One of our customers frequently encounters "serve-DataFrame" java.net.SocketTimeoutException: Accept timed errors in PySpark because DataSet.collectToPython() in Spark 2.4 does the following:
- Collects the results
- Opens up a socket server that is then listening to the connection from Python side
- Runs the event listeners as part of withAction on the same thread as
SPARK-25680is not available in Spark 2.4 - Returns the address of the socket server to Python
- The Python side connects to the socket server and fetches the data
As the customer has a custom, long running event listener the time between 2. and 5. is frequently longer than the default connection timeout and increasing the connect timeout is not a good solution as we don't know how long running the listeners can take.