Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-3522

A tablet server starts in non-functional state when enabling data-at-rest encryption

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.16.0, 1.17.0
    • 1.18.0
    • security, tserver
    • None

    Description

      It's possible to configure a Kudu tablet server by enabling the data-at-rest encryption feature in such a way that the server runs in a non-functional state: kudu-tserver process starts and runs with no visible issues, but it's not able to host any tablet replicas.

      It's easy to fix/address the issue by adding an extra sanity check: when opening an already existing FS data directory structure, make sure the server encryption key isn't empty if Kudu server is run with the --encrypt_data_at_rest flag. There might be more alternatives around.

      The reproduction scenario for the issue is below.

      1. Start a tablet server without encryption-at-rest, making sure the tablet server starts and creates the directory structure on the file system.
      2. Don't create any tables/ranges yet. Essentially, it's necessary to make sure not a single tablet replica is placed at the server yet.
      3. Shut down the tablet server.
      4. Update the configuration for the tablet server, enabling encryption-at-rest and specifying the key provider. For test purposes, it's enough to use the "default" key provider:
        --encrypt_data_at_rest=true
        --encryption_key_provider=default
        
      5. Start the tablet server.
      6. Try to create a new tablet replica that would be placed at the tablet server. That could be creation of a new table, or try to move a tablet replica from some other tablet server by using the kudu tablet change_config move_replica CLI tool.
      7. Check logs of Kudu master or the kudu CLI tool: there should be error messages like Failed to initialize encryption: error:0607B083:digital envelope routines:EVP_CipherInit_ex:no cipher set
      8. No tablet replica can now be placed at the tablet server, while nothing suspicious can be found in the tablet server's log.

      Attachments

        Activity

          People

            abukor Attila Bukor
            aserbin Alexey Serbin
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: