Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-32429

Standalone Mode allow setting CUDA_VISIBLE_DEVICES on executor launch

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.0.0
    • None
    • Deploy
    • None

    Description

      It would be nice if standalone mode could allow users to set CUDA_VISIBLE_DEVICES before launching an executor.  This has multiple benefits. 

      • kind of an isolation in that the executor can only see the GPUs set there. 
      • If your GPU application doesn't support explicitly setting the GPU device id, setting this will make any GPU look like the default (id 0) and things generally just work without any explicit setting
      • New features are being added on newer GPUs that require explicit setting of CUDA_VISIBLE_DEVICES like MIG (https://www.nvidia.com/en-us/technologies/multi-instance-gpu/)

      The code changes to just set this are very small, once we set them we would also possibly need to change the gpu addresses as it changes them to start from device id 0 again.

      The easiest implementation would just specifically support this and have it behind a config and set when the config is on and GPU resources are allocated. 

      Note we probably want to have this same thing set when we launch a python process as well so that it gets same env.

      Attachments

        Activity

          People

            Unassigned Unassigned
            tgraves Thomas Graves
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: