Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-20327

Add CLI support for YARN custom resources, like GPUs

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.1.0
    • 3.0.0
    • Spark Shell, Spark Submit

    Description

      YARN-3926 adds the ability for administrators to configure custom resources, like GPUs. This JIRA is to add support to Spark for requesting resources other than CPU virtual cores and memory. See YARN-3926.

      Attachments

        Issue Links

          Dependent

          Sub-task - The sub-task of the issue SPARK-23151 Provide a distribution of Spark with Hadoop 3.0

          • Major - Major loss of function.
          • Resolved
          depends upon

          New Feature - A new feature of the product, which has yet to be developed. YARN-3926 [Umbrella] Extend the YARN resource model for easier resource-type management and profiles

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #20761 [Github] Pull Request #20761 (szyszy)

          Pull request #22751 [Github] Pull Request #22751 (vanzin)

          Pull request #22751 [Github] Pull Request #22751 (vanzin)

          Activity

            People

              snemeth Szilard Nemeth
              templedf Daniel Templeton
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: