Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-7544

Transparent table/CF encryption

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.98.0
    • HFile, io
    • None
    • Reviewed
    • This change introduces a transparent encryption feature for protecting HFile and WAL data at rest. For detailed information including configuration examples see the Security section of the HBase manual.

    Description

      Introduce transparent encryption of HBase on disk data.

      Depends on a separate contribution of an encryption codec framework to Hadoop core and an AES-NI (native code) codec. This is work done in the context of MAPREDUCE-4491 but I'd gather there will be additional JIRAs for common and HDFS parts of it.

      Requirements:

      • Transparent encryption at the CF or table level
      • Protect against all data leakage from files at rest
      • Two-tier key architecture for consistency with best practices for this feature in the RDBMS world
      • Built-in key management
      • Flexible and non-intrusive key rotation
      • Mechanisms not exposed to or modifiable by users
      • Hardware security module integration (via Java KeyStore)
      • HBCK support for transparently encrypted files (+ plugin architecture for HBCK)

      Additional goals:

      • Shell support for administrative functions
      • Avoid performance impact for the null crypto codec case
      • Play nicely with other changes underway: in HFile, block coding, etc.

      We're aiming for rough parity with Oracle's transparent tablespace encryption feature, described in http://www.oracle.com/technetwork/database/owp-security-advanced-security-11gr-133411.pdf as

      “Transparent Data Encryption uses a 2-tier key architecture for flexible and non-intrusive key rotation and least operational and performance impact: Each application table with at least one encrypted column has its own table key, which is applied to all encrypted columns in that table. Equally, each encrypted tablespace has its own tablespace key. Table keys are stored in the data dictionary of the database, while tablespace keys are stored in the header of the tablespace and additionally, the header of each underlying OS file that makes up the tablespace. Each of these keys is encrypted with the TDE master encryption key, which is stored outside of the database in an external security module: either the Oracle Wallet (a PKCS#12 formatted file that is encrypted using a passphrase supplied either by the designated security administrator or DBA during setup), or a Hardware Security Module (HSM) device for higher assurance […]

      Further design details forthcoming in a design document and patch as soon as we have all of the clearances in place.

      Attachments

        1. latency-single.7544.xlsx
          121 kB
          Andrew Kyle Purtell
        2. historical-shell.patch
          4 kB
          Andrew Kyle Purtell
        3. historical-7544.pdf
          1.06 MB
          Andrew Kyle Purtell
        4. historical-7544.patch
          259 kB
          Andrew Kyle Purtell
        5. 7544p4.patch
          54 kB
          Andrew Kyle Purtell
        6. 7544p3.patch
          68 kB
          Andrew Kyle Purtell
        7. 7544p3.patch
          93 kB
          Andrew Kyle Purtell
        8. 7544p2.patch
          38 kB
          Andrew Kyle Purtell
        9. 7544p2.patch
          43 kB
          Andrew Kyle Purtell
        10. 7544p1.patch
          79 kB
          Andrew Kyle Purtell
        11. 7544p1.patch
          81 kB
          Andrew Kyle Purtell
        12. 7544-final.patch
          309 kB
          Andrew Kyle Purtell
        13. 7544-addendum-1.patch
          0.6 kB
          Andrew Kyle Purtell
        14. 7544.patch
          259 kB
          Andrew Kyle Purtell
        15. 7544.patch
          267 kB
          Andrew Kyle Purtell
        16. 7544.patch
          314 kB
          Andrew Kyle Purtell
        17. 7544.patch
          313 kB
          Andrew Kyle Purtell
        18. 7544.patch
          312 kB
          Andrew Kyle Purtell
        19. 7544.patch
          313 kB
          Andrew Kyle Purtell

        Issue Links

          Activity

            People

              apurtell Andrew Kyle Purtell
              apurtell Andrew Kyle Purtell
              Votes:
              0 Vote for this issue
              Watchers:
              33 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: