Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-41550 Dynamic Allocation on K8S GA
  3. SPARK-39324

Log ExecutorDecommission as INFO level in TaskSchedulerImpl

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.0
    • 3.4.0
    • Spark Core
    • None

    Description

      Like the other module, `TaskSchedulerImpl` should log the decommission as `INFO` level.

      22/05/28 01:25:28 INFO KubernetesClusterSchedulerBackend: Decommission executors: 8
      22/05/28 01:25:28 INFO KubernetesClusterSchedulerBackend: Notify executor 8 to decommissioning.
      22/05/28 01:25:28 INFO BlockManagerMasterEndpoint: Mark BlockManagers (BlockManagerId(8, 100.103.40.13, 43353, None)) as being decommissioning.
      22/05/28 01:25:29 ERROR TaskSchedulerImpl: Lost executor 8 on 100.103.40.13: Executor decommission.
      22/05/28 01:25:29 INFO ExecutorMonitor: Executor 8 is removed. Remove reason statistics: ...
      22/05/28 01:25:29 INFO DAGScheduler: Executor lost: 8 (epoch 7)
      22/05/28 01:25:29 INFO BlockManagerMasterEndpoint: Trying to remove executor 8 from BlockManagerMaster.
      22/05/28 01:25:29 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(8, 100.103.40.13, 43353, None)
      22/05/28 01:25:29 INFO BlockManagerMaster: Removed 8 successfully in removeExecutor
      22/05/28 01:25:29 INFO DAGScheduler: Shuffle files lost for executor: 8 (epoch 7)
      22/05/28 01:25:34 INFO BlockManagerMaster: Removal of executor 8 requested
      22/05/28 01:25:34 INFO BlockManagerMasterEndpoint: Trying to remove executor 8 from BlockManagerMaster.
      22/05/28 01:25:34 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asked to remove non-existent executor 8
      

      Attachments

        Activity

          People

            dongjoon Dongjoon Hyun
            dongjoon Dongjoon Hyun
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: