Support plain Kerberos Authentication with Spark

Spark should work with plain Kerberos authentication. Currently, Spark can work with Hadoop delegation tokens, but not plain Kerberos. Hadoop's UserGroupInformation(UGI) class is responsible for handling security authentication in Spark. This UserGroupInformation(UGI) has support for Kerberos authentication, as well as Token authentication. Since Spark does not work correctly with the Kerberos auth method, it leads to a gap in fully supporting all the security authentication mechanisms.

If Kerberos is used to login in UserGroupInformation(UGI) using keytabs at the startup of drivers and executors, then Spark does not allow this logged-in UserGroupInformation(UGI) user to correctly propagate. The exception arises from the implementation of the runAsSparkUser method in SparkHadoopUtil.

The runAsSparkUser method in SparkHadoopUtil creates a new UGI based on the current static UGI and then transfers credentials from this current static UGI to the new UGI. This works well with other auth methods, except Kerberos. Transfer credentials implementation is not conducive for Kerberos auth model since it does not transfer all the required internal state of UGI( such as isKeytab and isKrbTkt). For Kerberos, the UGI has to be created from UGI.loginUserFromKeytab method only and not simply by doing a transfer credentials from the previous UGI to the new UGI.

Ideally, the CoarseGrainedExecutorBackend should login using keytab, similar to MesosCoarseGrainedExecutorBackend.