Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6414

Spark driver failed with NPE on job cancelation

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 1.2.1, 1.3.0
    • 1.2.2, 1.3.1, 1.4.0
    • Scheduler, Spark Core
    • None

    Description

      When a job group is cancelled, we scan through all jobs to determine which are members of the group. This scan assumes that the job group property is always set. If 'properties' is null in an active job, you get an NPE.

      We just need to make sure we ignore ones where activeJob.properties is null. We should also make sure it works if the particular property is missing.

      https://github.com/apache/spark/blob/branch-1.3/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L678

      Attachments

        Activity

          People

            hunglin Hung Lin
            ymakhno Yuri Makhno
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: