Details
-
New Feature
-
Status: Resolved
-
Minor
-
Resolution: Duplicate
-
2.3.0
-
None
-
None
Description
If GPU resources are required for executor PODS in a multi-tentan k8s cluster, the resource limits section needs to specify that GPUs are required [1]. This is a simple request:
resources:
limits:
nvidia.com/gpu: 1 # requesting 1 GPU
A simple configuration limit could be specified (just like executor memory):
--conf spark.kubernetes.executor.limit.gpus=1
[1] https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/
I have an implementation I will submit via a pull request.
Attachments
Issue Links
- duplicates
-
SPARK-24434 Support user-specified driver and executor pod templates
- Resolved
- links to