Description
It's possible to configure a Kudu tablet server by enabling the data-at-rest encryption feature in such a way that the server runs in a non-functional state: kudu-tserver process starts and runs with no visible issues, but it's not able to host any tablet replicas.
It's easy to fix/address the issue by adding an extra sanity check: when opening an already existing FS data directory structure, make sure the server encryption key isn't empty if Kudu server is run with the --encrypt_data_at_rest flag. There might be more alternatives around.
The reproduction scenario for the issue is below.
- Start a tablet server without encryption-at-rest, making sure the tablet server starts and creates the directory structure on the file system.
- Don't create any tables/ranges yet. Essentially, it's necessary to make sure not a single tablet replica is placed at the server yet.
- Shut down the tablet server.
- Update the configuration for the tablet server, enabling encryption-at-rest and specifying the key provider. For test purposes, it's enough to use the "default" key provider:
--encrypt_data_at_rest=true --encryption_key_provider=default
- Start the tablet server.
- Try to create a new tablet replica that would be placed at the tablet server. That could be creation of a new table, or try to move a tablet replica from some other tablet server by using the kudu tablet change_config move_replica CLI tool.
- Check logs of Kudu master or the kudu CLI tool: there should be error messages like Failed to initialize encryption: error:0607B083:digital envelope routines:EVP_CipherInit_ex:no cipher set
- No tablet replica can now be placed at the tablet server, while nothing suspicious can be found in the tablet server's log.