Details
-
Bug
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
3.1.2
-
None
-
$ kubectl version Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.15", GitCommit:"73dd5c840662bb066a146d0871216333181f4b64", GitTreeState:"clean", BuildDate:"2021-01-13T13:22:41Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.19-gke.1701", GitCommit:"d7cecefb99b58e8968f59b59d76448eb1e6ea403", GitTreeState:"clean", BuildDate:"2021-06-23T21:51:59Z", GoVersion:"go1.13.15b4", Compiler:"gc", Platform:"linux/amd64"} $ spark-submit --version version 3.1.2 Using Scala version 2.12.10, OpenJDK 64-Bit Server VM, 11.0.10$ kubectl version Client Version: version.Info{Major: "1" , Minor: "18" , GitVersion: "v1.18.15" , GitCommit: "73dd5c840662bb066a146d0871216333181f4b64" , GitTreeState: "clean" , BuildDate: "2021-01-13T13:22:41Z" , GoVersion: "go1.13.15" , Compiler : "gc" , Platform: "darwin/amd64" } Server Version: version.Info{Major: "1" , Minor: "18+" , GitVersion: "v1.18.19-gke.1701" , GitCommit: "d7cecefb99b58e8968f59b59d76448eb1e6ea403" , GitTreeState: "clean" , BuildDate: "2021-06-23T21:51:59Z" , GoVersion: "go1.13.15b4" , Compiler : "gc" , Platform: "linux/amd64" } $ spark-submit --version version 3.1.2 Using Scala version 2.12.10, OpenJDK 64-Bit Server VM, 11.0.10
Description
When connecting to a Google Kubernetes Engine, a command gcloud container clusters get-credentials is used that generates a ~/.kube/config file. The distinctive trait in this config file is that it uses an auth-provider relying on gcloud to inject the keys expiry and access-token from the general Google SDK auth config, as seen here:
users:
- name: gke_my-project_my-region_my-cluster
user:
auth-provider:
config:
cmd-args: config config-helper --format=json
cmd-path: /Users/user/google-cloud-sdk/bin/gcloud
expiry-key: '{.credential.token_expiry}'
token-key: '{.credential.access_token}'
kubectl, because it uses client-go, supports the auth-provider and fetches the token and expiry from the json returne by config-helper. As Spark is using the fabric8 client, this is yet to be supported, breaking when running spark-submit:
Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://my-endpoint/api/v1/namespaces/my-namespace/pods. Message: Forbidden! User gke_my-project_my-region_my-cluster doesn't have permission. pods is forbidden: User "system:anonymous" cannot create resource "pods" in API group "" in the namespace "my-namespace".