Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-20624 SPIP: Add better handling for node shutdown
  3. SPARK-36543

Decommission logs too frequent when waiting migration to finish

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: In Progress
    • Major
    • Resolution: Unresolved
    • 3.1.0, 3.2.0, 3.3.0
    • None
    • Spark Core
    • None

    Description

      21/08/18 08:14:31 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
      21/08/18 08:14:31 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
      21/08/18 08:14:31 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
      21/08/18 08:14:32 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
      21/08/18 08:14:32 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
      21/08/18 08:14:32 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
      21/08/18 08:14:33 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
      21/08/18 08:14:33 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
      21/08/18 08:14:33 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
      21/08/18 08:14:34 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
      21/08/18 08:14:34 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
      21/08/18 08:14:34 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
      21/08/18 08:14:35 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
      21/08/18 08:14:35 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
      21/08/18 08:14:35 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
      21/08/18 08:14:36 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
      21/08/18 08:14:36 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
      21/08/18 08:14:36 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
      21/08/18 08:14:37 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
      21/08/18 08:14:37 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
      21/08/18 08:14:37 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
      21/08/18 08:14:38 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
      21/08/18 08:14:38 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
      21/08/18 08:14:38 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
      21/08/18 08:14:39 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
      21/08/18 08:14:39 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
      21/08/18 08:14:39 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
      21/08/18 08:14:40 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
      21/08/18 08:14:40 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
      21/08/18 08:14:40 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
      21/08/18 08:14:41 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
      21/08/18 08:14:41 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
      21/08/18 08:14:41 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
      21/08/18 08:14:42 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
      21/08/18 08:14:42 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
      21/08/18 08:14:42 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
      21/08/18 08:14:43 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
      21/08/18 08:14:43 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
      21/08/18 08:14:43 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
      21/08/18 08:14:44 INFO CoarseGrainedExecutorBackend: Checking to see if we can shutdown.
      21/08/18 08:14:44 INFO CoarseGrainedExecutorBackend: No running tasks, checking migrations
      21/08/18 08:14:44 INFO CoarseGrainedExecutorBackend: All blocks not yet migrated.
      ...

      It takes some time to migrate data (shuffle or rdd). Logging per second is too frequent. 

      Attachments

        Activity

          People

            apachespark Apache Spark
            Ngone51 wuyi
            Holden Karau Holden Karau
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated: