Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27823

Add an abstraction layer for accelerator resource handling to avoid manipulating raw confs

    XMLWordPrintableJSON

Details

    • Story
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • Spark Core
    • None

    Description

      In SPARK-27488, we extract resource requests and allocation by parsing raw Spark confs. This hurts readability because we didn't have the abstraction at resource level. After we merge the core changes, we should do a refactoring and make the code more readable.

      See https://github.com/apache/spark/pull/24615#issuecomment-494580663.

      Attachments

        Issue Links

          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. SPARK-27366 Spark scheduler internal changes to support GPU scheduling

          • Major - Major loss of function.
          • Resolved

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. SPARK-27488 Driver interface to support GPU resources

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #24821

          Web Link GitHub Pull Request #24856

          Web Link GitHub Pull Request #24954

          Activity

            People

              mengxr Xiangrui Meng
              mengxr Xiangrui Meng
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: