Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27963

Allow dynamic allocation without an external shuffle service

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • Spark Core

    Description

      It would be useful for users to be able to enable dynamic allocation without the need to provision an external shuffle service. One immediate use case is the ability to use dynamic allocation on Kubernetes, which doesn't yet have that service.

      This has been suggested before (e.g. https://github.com/apache/spark/pull/24083, which was attached to the k8s-specific SPARK-24432), and can actually be done without affecting the internals of the Spark scheduler (aside from the dynamic allocation code).

      Attachments

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-31645 Improve ExecutorMonitor shuffle-tracking to handle speculative tasks properly

          • Major - Major loss of function.
          • Open
          relates to

          New Feature - A new feature of the product, which has yet to be developed. SPARK-24432 Add support for dynamic resource allocation

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-28487 K8S pod allocator behaves poorly with dynamic allocation

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #28452 [Github] Pull Request #28452 (dongjoon-hyun)

          Pull request #28452 [Github] Pull Request #28452 (dongjoon-hyun)

          Web Link GitHub Pull Request #24817

          Web Link GitHub Pull Request #28132

          Activity

            People

              vanzin Marcelo Masiero Vanzin
              vanzin Marcelo Masiero Vanzin
              Votes:
              0 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: