Details
-
Improvement
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
3.1.0
-
None
Description
Application log:
Failed to process batch org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [800 seconds]. This timeout is controlled by spark.network.timeout: org.apache.spark.rpc.RpcTimeout.org $apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:47)
Driver log:
[block-manager-ask-thread-pool-149] WARN org.apache.spark.storage.BlockManagerMaster - Failed to remove RDD 25344 - Cannot receive any reply from null in 800 seconds. This timeout is controlled by spark.network.timeout org.apache.spark.rpc.RpcTimeoutException: Cannot receive any reply from null in 800 seconds. This timeout is controlled by spark.network.timeout
The log message does not provide RpcAddress of the destination RpcEndpoint. It is due to
* The `rpcAddress` may be null, in which case the endpoint is registered via a client-only * connection and can only be reached via the client that sent the endpoint reference.
Solution:
using rpcAdress from client of the NettyRpcEndpoingRef once such endpoint resides in client mode.