Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-29340

Spark Sql executions do not use thread local jobgroup

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Not A Problem
    • 2.4.4
    • None
    • Spark Core
    • None

    Description

      val sparkThreadLocal: SparkSession = DataCurator.spark.newSession()

      sparkThreadLocal.sparkContext.setJobGroup("<id>", "<description>")

      OR

      sparkThreadLocal.sparkContext.setLocalProperty("spark.job.description", "<id>")
      sparkThreadLocal.sparkContext.setLocalProperty("spark.jobGroup.id", "<description>")

       

      The jobgroup property works fine for spark jobs/stages created by spark dataframe operations but in case of sparksql the jobgroup is randomly assigned to stages or is null sometimes.

      Attachments

        Activity

          People

            Unassigned Unassigned
            navdeepniku Navdeep Poonia
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: