Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-43912

Incorrect SparkException for Stage-Level Scheduling in local mode

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 3.4.0
    • None
    • Scheduler
    • None
    • ```text
      scala> println(spark.version)
      3.4.0

      scala> println(sc.master)
      local[*]
      ```

    Description

      While in `local[*]` mode, the following `SparkException` is thrown:

      ```text
      org.apache.spark.SparkException: TaskResourceProfiles are only supported for Standalone cluster for now when dynamic allocation is disabled.
      at org.apache.spark.resource.ResourceProfileManager.isSupported(ResourceProfileManager.scala:71)
      at org.apache.spark.resource.ResourceProfileManager.addResourceProfile(ResourceProfileManager.scala:126)
      at org.apache.spark.rdd.RDD.withResources(RDD.scala:1802)
      ... 42 elided
      ```

      This happens for the following snippet:

      ```scala
      val rdd = sc.range(0, 9)

      import org.apache.spark.resource.ResourceProfileBuilder
      val rpb = new ResourceProfileBuilder
      val rp1 = rpb.build()

      rdd.withResources(rp1)
      ```

      Attachments

        Activity

          People

            Unassigned Unassigned
            jlaskowski Jacek Laskowski
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: