Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-44862

Adding metric tracking number of jobs running on a cluster

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 4.0.0
    • None
    • Web UI
    • None

    Description

      We frequently come across issues where in there are considerable number of jobs/notebooks running on one interactive cluster. This leads to either degraded performance due to cluster being overloaded or leads to driver OOM. It would be helpful to have a metric identifying number of notebooks/jobs running on a cluster as this would be important datapoint for customers to balance out their workloads.

      We propose to add a metric which keeps track of number of jobs running on a cluster.

      Attachments

        1. image-2023-09-19-11-21-55-416.png
          38 kB
          Ruifeng Zheng

        Activity

          People

            yao Kent Yao
            podongfeng Ruifeng Zheng
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: