Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-41550

Dynamic Allocation on K8S GA

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.0
    • 3.4.0
    • Kubernetes

    Description

      In the Kubernetes environments, this issue aims to make `Dynamic Allocation` as GA in Apache Spark 3.4.

      Attachments

        Issue Links

          relates to

          New Feature - A new feature of the product, which has yet to be developed. SPARK-37810 Executor Rolling in Kubernetes environment

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. SPARK-41515 PVC-oriented executor pod allocation

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-20624 SPIP: Add better handling for node shutdown

          • Major - Major loss of function.
          • In Progress
          links to

          Web Link Cloud Shuffle Storage Plugin for Apache Spark

          1.
          		 Remove `Experimental` from `spark.dynamicAllocation.shuffleTracking.enabled` Sub-task Resolved Dongjoon Hyun
          2.
          		 Remove `Experimental` from `RDD.cleanShuffleDependencies` API Sub-task Resolved Dongjoon Hyun
          3.
          		 Remove `Dynamic Resource Allocation` from K8s Future Work Sub-task Resolved Dongjoon Hyun
          4.
          		 Enable spark.dynamicAllocation.shuffleTracking.enabled by default Sub-task Resolved Dongjoon Hyun
          5.
          		 Enable spark.storage.decommission.(rdd|shuffle)Blocks.enabled by default Sub-task Resolved Dongjoon Hyun
          6.
          		 Enable spark.storage.replication.proactive by default Sub-task Resolved Dongjoon Hyun
          7.
          		 Add PySpark cleanShuffleDependencies API Sub-task Resolved Dongjoon Hyun
          8.
          		 Reuse PVCs by default Sub-task Resolved Dongjoon Hyun
          9.
          		 ExecutorMonitor.timedOutExecutors should be deterministic Sub-task Resolved Dongjoon Hyun
          10.
          		 Log ExecutorDecommission as INFO level in TaskSchedulerImpl Sub-task Resolved Dongjoon Hyun
          11.
          		 Add decomTestTag to K8s Integration Test Sub-task Resolved Dongjoon Hyun
          12.
          		 Add `DISK_USED` executor roll policy Sub-task Resolved Dongjoon Hyun
          13.
          		 Add `TOTAL_SHUFFLE_WRITE` executor roll policy Sub-task Resolved Dongjoon Hyun
          14.
          		 Support PEAK_JVM_(ON|OFF)HEAP_MEMORY executor rolling policy Sub-task Resolved Dongjoon Hyun
          15.
          		 ExecutorMonitor.onExecutorRemoved should handle ExecutorDecommission as finished Sub-task Resolved Dongjoon Hyun
          16.
          		 Propagate decommission executor loss reason during onDisconnect in K8s Sub-task Resolved Holden Karau
          17.
          		 Keep removed executor info in decommission state Sub-task Resolved Zhongwei Zhu
          18.
          		 Populate ExecutorDecommission with more informative messages Sub-task Resolved Bo Zhang
          19.
          		 Fix wrong remained shuffles log in BlockManagerDecommissioner Sub-task Resolved Zhongwei Zhu
          20.
          		 Ignore stage fetch failure caused by decommissioned executor Sub-task Resolved Zhongwei Zhu
          21.
          		 Randomize the orders of peer in BlockManagerDecommissioner Sub-task Resolved Zhongwei Zhu
          22.
          		 Add numberDecommissioningExecutors metric Sub-task Resolved Zhongwei Zhu
          23.
          		 FallbackStorage shouldn't attempt to resolve arbitrary "remote" hostname Sub-task Resolved Erik Krogen
          24.
          		 Support `local[*]` in `spark-submit` in K8s environment Sub-task Resolved Dongjoon Hyun
          25.
          		 Log `exitCode` when `SparkContext.stop` starts Sub-task Resolved Dongjoon Hyun
          26.
          		 Ignore `IOException` in `handleBlockRemovalFailure` if SparkContext is stopped Sub-task Resolved Dongjoon Hyun

          Activity

            People

              dongjoon Dongjoon Hyun
              dongjoon Dongjoon Hyun
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: