Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-10768

Optimize Hadoop RPC encryption performance

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • 3.0.0-alpha1
    • None
    • performance, security
    • None
    • Hide
      This improvement leverages AES-NI hardware acceleration to optimize Hadoop RPC performance.

      To enable it, adding the following configuration in core-site.xml
      <property>
      <name>hadoop.rpc.security.crypto.cipher.suites</name>
      <value>AES/CTR/NoPadding</value>
      </property>

      Optionally, set key bit length (default 128) via configuration key hadoop.rpc.security.crypto.cipher.key.bitlength in core-site.xml.
      Show
      This improvement leverages AES-NI hardware acceleration to optimize Hadoop RPC performance. To enable it, adding the following configuration in core-site.xml <property> <name>hadoop.rpc.security.crypto.cipher.suites</name> <value>AES/CTR/NoPadding</value> </property> Optionally, set key bit length (default 128) via configuration key hadoop.rpc.security.crypto.cipher.key.bitlength in core-site.xml.

    Description

      Hadoop RPC encryption is enabled by setting hadoop.rpc.protection to "privacy". It utilized SASL GSSAPI and DIGEST-MD5 mechanisms for secure authentication and data protection. Even GSSAPI supports using AES, but without AES-NI support by default, so the encryption is slow and will become bottleneck.

      After discuss with atm, tucu00 and umamaheswararao, we can do the same optimization as in HDFS-6606. Use AES-NI with more than 20x speedup.

      On the other hand, RPC message is small, but RPC is frequent and there may be lots of RPC calls in one connection, we needs to setup benchmark to see real improvement and then make a trade-off.

      Attachments

        1. cpu_profile_RPC_encryption_AES.png
          373 kB
          Wei-Chiu Chuang
        2. cpu_profile_rpc_encryption_optimize_calculateHMAC.png
          394 kB
          Wei-Chiu Chuang
        3. HADOOP-10768.001.patch
          71 kB
          Dian Fu
        4. HADOOP-10768.002.patch
          76 kB
          Dian Fu
        5. HADOOP-10768.003.patch
          78 kB
          Aaron Myers
        6. HADOOP-10768.004.patch
          121 kB
          Dapeng Sun
        7. HADOOP-10768.005.patch
          94 kB
          Dapeng Sun
        8. HADOOP-10768.006.patch
          95 kB
          Dapeng Sun
        9. HADOOP-10768.007.patch
          94 kB
          Dapeng Sun
        10. HADOOP-10768.008.patch
          94 kB
          Dapeng Sun
        11. HADOOP-10768.009.patch
          99 kB
          Wei-Chiu Chuang
        12. HADOOP-10768.010.patch
          95 kB
          Wei-Chiu Chuang
        13. HADOOP-10768.011.patch
          99 kB
          Wei-Chiu Chuang
        14. Optimize Hadoop RPC encryption performance.pdf
          280 kB
          Dian Fu

        Issue Links

          Activity

            People

              weichiu Wei-Chiu Chuang
              hitliuyi Yi Liu
              Votes:
              0 Vote for this issue
              Watchers:
              56 Start watching this issue

              Dates

                Created:
                Updated: