Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-41550

Dynamic Allocation on K8S GA

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.0
    • 3.4.0
    • Kubernetes

    Description

      In the Kubernetes environments, this issue aims to make `Dynamic Allocation` as GA in Apache Spark 3.4.

      Attachments

        Issue Links

        relates to

        New Feature - A new feature of the product, which has yet to be developed. SPARK-37810 Executor Rolling in Kubernetes environment

        • Major - Major loss of function.
        • Resolved

        New Feature - A new feature of the product, which has yet to be developed. SPARK-41515 PVC-oriented executor pod allocation

        • Major - Major loss of function.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. SPARK-20624 SPIP: Add better handling for node shutdown

        • Major - Major loss of function.
        • In Progress
        links to

        Web Link Cloud Shuffle Storage Plugin for Apache Spark

        1.
        		 Remove `Experimental` from `spark.dynamicAllocation.shuffleTracking.enabled` Sub-task Resolved Dongjoon Hyun Actions
        2.
        		 Remove `Experimental` from `RDD.cleanShuffleDependencies` API Sub-task Resolved Dongjoon Hyun Actions
        3.
        		 Remove `Dynamic Resource Allocation` from K8s Future Work Sub-task Resolved Dongjoon Hyun Actions
        4.
        		 Enable spark.dynamicAllocation.shuffleTracking.enabled by default Sub-task Resolved Dongjoon Hyun Actions
        5.
        		 Enable spark.storage.decommission.(rdd|shuffle)Blocks.enabled by default Sub-task Resolved Dongjoon Hyun Actions
        6.
        		 Enable spark.storage.replication.proactive by default Sub-task Resolved Dongjoon Hyun Actions
        7.
        		 Add PySpark cleanShuffleDependencies API Sub-task Resolved Dongjoon Hyun Actions
        8.
        		 Reuse PVCs by default Sub-task Resolved Dongjoon Hyun Actions
        9.
        		 ExecutorMonitor.timedOutExecutors should be deterministic Sub-task Resolved Dongjoon Hyun Actions
        10.
        		 Log ExecutorDecommission as INFO level in TaskSchedulerImpl Sub-task Resolved Dongjoon Hyun Actions
        11.
        		 Add decomTestTag to K8s Integration Test Sub-task Resolved Dongjoon Hyun Actions
        12.
        		 Add `DISK_USED` executor roll policy Sub-task Resolved Dongjoon Hyun Actions
        13.
        		 Add `TOTAL_SHUFFLE_WRITE` executor roll policy Sub-task Resolved Dongjoon Hyun Actions
        14.
        		 Support PEAK_JVM_(ON|OFF)HEAP_MEMORY executor rolling policy Sub-task Resolved Dongjoon Hyun Actions
        15.
        		 ExecutorMonitor.onExecutorRemoved should handle ExecutorDecommission as finished Sub-task Resolved Dongjoon Hyun Actions
        16.
        		 Propagate decommission executor loss reason during onDisconnect in K8s Sub-task Resolved Holden Karau Actions
        17.
        		 Keep removed executor info in decommission state Sub-task Resolved Zhongwei Zhu Actions
        18.
        		 Populate ExecutorDecommission with more informative messages Sub-task Resolved Bo Zhang Actions
        19.
        		 Fix wrong remained shuffles log in BlockManagerDecommissioner Sub-task Resolved Zhongwei Zhu Actions
        20.
        		 Ignore stage fetch failure caused by decommissioned executor Sub-task Resolved Zhongwei Zhu Actions
        21.
        		 Randomize the orders of peer in BlockManagerDecommissioner Sub-task Resolved Zhongwei Zhu Actions
        22.
        		 Add numberDecommissioningExecutors metric Sub-task Resolved Zhongwei Zhu Actions
        23.
        		 FallbackStorage shouldn't attempt to resolve arbitrary "remote" hostname Sub-task Resolved Erik Krogen Actions
        24.
        		 Support `local[*]` in `spark-submit` in K8s environment Sub-task Resolved Dongjoon Hyun Actions
        25.
        		 Log `exitCode` when `SparkContext.stop` starts Sub-task Resolved Dongjoon Hyun Actions
        26.
        		 Ignore `IOException` in `handleBlockRemovalFailure` if SparkContext is stopped Sub-task Resolved Dongjoon Hyun Actions

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            dongjoon Dongjoon Hyun
            dongjoon Dongjoon Hyun
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment