Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-45869

Revisit and Improve Spark Standalone Cluster

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 4.0.0
    • 4.0.0
    • Spark Core

    Description

      Spark Standalone Cluster has been supported for a long time as one of the resource managers.

      As a part of Apache Spark 4.0.0, we revisit all layers of `Spark Standalone Cluster` as a long running subsystem inside K8s environment.

      1. Spark Master, Worker, History Server Web UI Layer
      2. Spark Master HA and Recovery Layer
      3. Spark Master REST API Layer (including Cluster Utilization monitoring)
      4. Spark Job Scheduling Layer
      5. Spark Worker Management by exposing Cluster Utilization monitoring for Elastic Cluster Management
      6. Spark Master/Worker dependency and classpath audit
      7. Documentation

      Attachments

        Issue Links

          There are no Sub-Tasks for this issue.

          Activity

            People

              dongjoon Dongjoon Hyun
              dongjoon Dongjoon Hyun
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: