Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-16136

Handle all occurrence of InvalidEncryptionKeyException

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.1.1
    • None
    • None
    • None

    Description

      After HDFS-10609, HDFS-11741, we still observed InvalidEncryptionKeyException errors that are not retried.

      2021-07-12 11:10:58,795 ERROR datanode.DataNode (DataXceiver.java:writeBlock(863)) - DataNode{data=FSDataset{dirpath='[/grid/01/hadoop/hdfs/data, /grid/02/hadoop/hdfs/data, /grid/03/hadoop/hdfs/data, /grid/04/hadoop/hdfs/data, /grid/05/hadoop/hdfs/data, /grid/06/hadoop/hdfs/data, /grid/07/hadoop/hdfs/data, /grid/08/hadoop/hdfs/data, /grid/09/hadoop/hdfs/data, /grid/10/hadoop/hdfs/data, /grid/11/hadoop/hdfs/data, /grid/12/hadoop/hdfs/data, /grid/13/hadoop/hdfs/data, /grid/14/hadoop/hdfs/data, /grid/15/hadoop/hdfs/data, /grid/16/hadoop/hdfs/data, /grid/17/hadoop/hdfs/data, /grid/18/hadoop/hdfs/data, /grid/19/hadoop/hdfs/data, /grid/20/hadoop/hdfs/data, /grid/21/hadoop/hdfs/data, /grid/22/hadoop/hdfs/data]'}, localName='lxdmelcly-lxw01-p01-whw10289.oan:10019', datanodeUuid='70403b64-cb39-4b4a-ac6c-787ce7bdbe2c', xmitsInProgress=0}:Exception transfering block BP-1743446178-172.18.16.38-1537373339905:blk_2196991498_1131235321 to mirror 172.18.16.33:10019
org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException: Can't re-compute encryption key for nonce, since the required block key (keyID=-213389155) doesn't exist. Current key: 1804780309
        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.readSaslMessageAndNegotiatedCipherOption(DataTransferSaslUtil.java:419)
        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:479)
        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:303)
        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:245)
        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:215)
        at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.socketSend(SaslDataTransferClient.java:183)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:800)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:173)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:107)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290)
        at java.lang.Thread.run(Thread.java:745)
2021-07-12 11:10:58,796 ERROR datanode.DataNode (DataXceiver.java:run(321)) - xxx:10019:DataXceiver error processing WRITE_BLOCK operation  src: /172.18.16.8:41992 dst: /172.18.16.20:10019
org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException: Can't re-compute encryption key for nonce, since the required block key (keyID=-213389155) doesn't exist. Current key: 1804780309

      We should handle this exception whenever SaslDataTransferClient.socketSend() is invoked:

      DataXceiver.writeBlock()
      BlockDispatcher.moveBlock()
      DataNode.run()
      DataXceiver.replaceBlock()
      StripedBlockWriter.init()

      This issue isn't that obvious, because the existing HDFS fault tolerance mechanisms should mask the data encryption key error.

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-10609 Uncaught InvalidEncryptionKeyException during pipeline recovery may abort downstream applications

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-11741 Long running balancer may fail due to expired DataEncryptionKey

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              weichiu Wei-Chiu Chuang
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: