Design client side SSL certificate verification in Libprocess.

Notes from an offline discussion with vinodkone, tillt, jgehrcke, CarlDellar.

• Authentication can happen at the transport and/or at the application layer. There is no real benefit in doing it at both layers.
• Authentication at the application layer allows for subsequent authorization.
• We would like to have an option to mutually authenticate all components in a Mesos cluster, including external tooling, regardless at which layer, to secure communication channels.
• Mutual authentication at the transport layer everywhere can be hard because some components can't or don't want to provide certificates, e.g., a Lua HTTP client reading master's state.
• Theoretically, some components, e.g., Mesos masters and agents, can form an ensemble inside which all connections are authenticated on both sides at the transport layer (TLS certificate verification). Practically, it may then be hard to implement communication with the components outside such ensemble, e.g., frameworks, executors, since at least two types of connections/sockets should be distinguished: with and without client certificate verification (Libprocess can't do it now), or all the traffic between the ensemble and outside components should go via a proxy.
• An alternative is to combine server side TLS certificate verification with the client side application layer authentication. For that to be secure, we need to implement client authentication for Mesos components, e.g., master with agent, replica with other replica (see MESOS-9638). Plus relax certificate verification option in Libprocess for outgoing connections only. For non-streaming connections a secret connection identifier should be passed by the client to prove they are the entity that has been previously authenticated.
• Whatever path we choose, truly secure communication channels will become when separate certificates for Mesos components are used, either signed by a different root CA or using a specific CN/SAN, which can't be obtained by everyone.

What needs to be done:

• Introduce or adjust the Libprocess flag for verifying certificates for outgoing connections only.
• Verify how replicas in the master's replicated log discover other replicas and what harm a rogue replica can do if it tries to join the quorum. Estimate whether master's replicated log can use its own copy of Libprocess.
• Implement Mesos master authentication with Mesos agents, MESOS-9638.