Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27394

The staleness of UI may last minutes or hours when no tasks start or finish

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.4.0, 2.4.1
    • 2.4.2, 3.0.0
    • Web UI
    • None

    Description

      Run the following codes on a cluster that has at least 2 cores.

      sc.makeRDD(1 to 1000, 1000).foreach { i =>
        Thread.sleep(300000)
      }
      

      The jobs page will just show one running task.

      This is because when the second task event calls "AppStatusListener.maybeUpdate" for a job, it will just ignore since the gap between two events is smaller than `spark.ui.liveUpdate.period`.

      After the second task event, in the above case, because there won't be any other task events, the Spark UI will be always stale until the next task event gets fired (after 300 seconds).

      Attachments

        Activity

          People

            zsxwing Shixiong Zhu
            zsxwing Shixiong Zhu
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: