Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31485

Barrier stage can hang if only partial tasks launched

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.4.0
    • 2.4.6
    • Spark Core
    • None

    Description

      The issue can be reproduced by following test:

       

      initLocalClusterSparkContext(2)
      val rdd0 = sc.parallelize(Seq(0, 1, 2, 3), 2)
      val dep = new OneToOneDependency[Int](rdd0)
      val rdd = new MyRDD(sc, 2, List(dep), Seq(Seq("executor_h_0"),Seq("executor_h_0")))
      rdd.barrier().mapPartitions { iter =>
        BarrierTaskContext.get().barrier()
        iter
      }.collect()
      

       

      Attachments

        Activity

          People

            Ngone51 wuyi
            Ngone51 wuyi
            Holden Karau Holden Karau
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: