Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27488

Driver interface to support GPU resources

    XMLWordPrintableJSON

Details

    • Story
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • Spark Core
    • None

    Description

      We want to have an interface to allow the users on the driver to get what resources are allocated to them.  This is mostly to handle the case the cluster manager does not launch the driver in an isolated environment and where users could be sharing hosts.  For instance in standalone mode it doesn't support container isolation so a host may have 4 gpu's, but only 2 of them could be assigned to the driver.  In this case we need an interface for the cluster manager to specify what gpu's for the driver to use and an interface for the user to get the resource information

      Attachments

        Issue Links

          blocks

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. SPARK-27823 Add an abstraction layer for accelerator resource handling to avoid manipulating raw confs

          • Major - Major loss of function.
          • Resolved
          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. SPARK-27024 Executor interface for cluster managers to support GPU resources

          • Major - Major loss of function.
          • Resolved
          is related to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. SPARK-27364 User-facing APIs for GPU-aware scheduling

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #24615

          Activity

            People

              tgraves Thomas Graves
              tgraves Thomas Graves
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: