Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
Impala 4.3.0
-
None
-
ghx-label-12
Description
A test cluster ran into issues with idle connections being disconnected when using SSL.
This reproduces on my development environment with these steps:
- Start Impala with SSL enabled
bin/start-impala-cluster.py --impalad_args="--ssl_client_ca_certificate=${IMPALA_HOME}/be/src/testutil/server-cert.pem --ssl_server_certificate=${IMPALA_HOME}/be/src/testutil/server-cert.pem --ssl_private_key=${IMPALA_HOME}/be/src/testutil/server-key.pem --hostname=localhost --idle_client_poll_period_s=30 -v=2" --state_store_args="--ssl_client_ca_certificate=${IMPALA_HOME}/be/src/testutil/server-cert.pem --ssl_server_certificate=${IMPALA_HOME}/be/src/testutil/server-cert.pem --ssl_private_key=${IMPALA_HOME}/be/src/testutil/server-key.pem --hostname=localhost" --catalogd_args="--ssl_client_ca_certificate=${IMPALA_HOME}/be/src/testutil/server-cert.pem --ssl_server_certificate=${IMPALA_HOME}/be/src/testutil/server-cert.pem --ssl_private_key=${IMPALA_HOME}/be/src/testutil/server-key.pem --hostname=localhost" --cluster_size=1
- Connect with impala-shell
impala-shell --ssl
- Leave this idle for 150+ seconds
In the Impalad logs will be a statement like this:
I0503 22:11:53.233147 206554 impala-server.cc:2488] Connection 20470cb275a1d256:3d68601942f3179f from client 172.27.100.70:42540 to server hiveserver2-frontend closed. The connection had 2 associated session(s).
- Run a statement in impala-shell and will show that it needs to reconnect
default> show tables; Caught exception TSocket read 0 bytes, type=<class 'thrift.transport.TTransport.TTransportException'> in PingImpalaHS2Service. Caught exception [Errno 32] Broken pipe, type=<class 'socket.error'> in CloseSession. Warning: close session RPC failed: [Errno 32] Broken pipe, <class 'socket.error'> Connection lost, reconnecting... ... then it retries and succeeds
Tracing through the code, it appears that this peek() call returns false:
try {
bytes_pending = input_->getTransport()->peek();
break;
} catch (const TTransportException& ttx) {
bytes_pending is false, and this causes the connection to be closed.
This doesn't seem to impact Impala with older Thrift versions, so maybe something changed in Thrift 0.16.
Attachments
Issue Links
- relates to
-
-
- Open
-