Details
-
Bug
-
Status: Patch Available
-
Major
-
Resolution: Unresolved
-
2.2.7
-
None
-
None
Description
We found this problem in our production. LogRoller stuck by FanOutOneBlockAsyncDFSOutputHelper.createOutput waitting get future all time
Check the regionserver's log, the regionServer do sasl negotiate with two dataNode, but just one check complete. Another do nothing after connected with dn.
518415 2023-04-17 14:17:25,434 INFO io.transwarp.guardian.client.cache.PeriodCacheUpdater: Fetch change version: 0 518416 2023-04-17 14:17:29,092 DEBUG org.apache.hadoop.hbase.ScheduledChore: RefreshCredentials execution time: 0 ms. 518417 2023-04-17 14:17:29,768 DEBUG org.apache.hadoop.hbase.ScheduledChore: CompactionChecker execution time: 0 ms. 518418 2023-04-17 14:17:29,768 DEBUG org.apache.hadoop.hbase.ScheduledChore: CompactionThroughputTuner execution time: 0 ms.518419 2023-04-17 14:17:29,768 DEBUG org.apache.hadoop.hbase.ScheduledChore: MemstoreFlusherChore execution time: 0 ms. 518420 2023-04-17 14:17:29,768 DEBUG org.apache.hadoop.hbase.ScheduledChore: gy-dmz-swrzjzcc-gx-2-19,60020,1677341424491-Hea pMemoryTunerChore execution time: 0 ms. 518421 2023-04-17 14:17:39,375 DEBUG org.apache.hadoop.hbase.regionserver.LogRoller: WAL AsyncFSWAL gy-dmz-swrzjzcc-gx-2-19% 2C60020%2C1677341424491:(num 1681711899342) roll requested 518422 2023-04-17 14:17:39,389 DEBUG org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper: SASL client doing general handshake for addr = 10.179.157.10/10.179.157.10, datanodeId = DatanodeInfoWithStorage[10.179.157.10:50 010,DS-4815c34a-8d0c-42b9-b56c-529d2732d956,DISK] 518423 2023-04-17 14:17:39,391 DEBUG org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper: SASL client doing general handshake for addr = 10.179.157.29/10.179.157.29, datanodeId = DatanodeInfoWithStorage[10.179.157.29:50 010,DS-509f84fe-2e88-403e-87b5-f4765e49094f,DISK] 518424 2023-04-17 14:17:39,392 DEBUG org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper: Verifying QO P, requested QOP = [auth], negotiated QOP = auth 518425 2023-04-17 14:17:39,743 DEBUG org.apache.hadoop.hbase.ScheduledChore: MemstoreFlusherChore execution time: 0 ms. 518426 2023-04-17 14:17:39,743 DEBUG org.apache.hadoop.hbase.ScheduledChore: CompactionChecker execution time: 0 ms. 518427 2023-04-17 14:17:49,977 DEBUG org.apache.hadoop.hbase.ScheduledChore: CompactionChecker execution time: 0 ms. 518428 2023-04-17 14:17:49,977 DEBUG org.apache.hadoop.hbase.ScheduledChore: MemstoreFlusherChore execution time: 0 ms. 518429 2023-04-17 14:17:55,492 INFO
FanOutOneBlockAsyncDFSOutputHelper.createOutput will connect and trySaslNegotiate to dataNode. In Sasl authentication mode, SaslNegotiateHandler will be used to handle authentication. If datanode is shut down, SaslNegotiateHandler.channelInactive do not call back to promise and cause future to be stuck forever.
@Override public void handlerAdded(ChannelHandlerContext ctx) throws Exception { ctx.write(ctx.alloc().buffer(4).writeInt(SASL_TRANSFER_MAGIC_NUMBER)); sendSaslMessage(ctx, new byte[0]); ctx.flush(); step++; } @Override public void channelInactive(ChannelHandlerContext ctx) throws Exception { saslClient.dispose(); }
So SaslNegotiateHandler.channelInactive should call promise.tryFailure to avoid future stuck forever.
