Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25727

makeCopy failed in InMemoryRelation

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.4.0
    • 2.4.0
    • SQL
    • None

    Description

          val data = Seq(100).toDF("count").cache()
          data.queryExecution.optimizedPlan.toJSON
      

      The above code can generate the following error:

      assertion failed: InMemoryRelation fields: output, cacheBuilder, statsOfPlanToCache, outputOrdering, values: List(count#178), CachedRDDBuilder(true,10000,StorageLevel(disk, memory, deserialized, 1 replicas),*(1) Project [value#176 AS count#178]
      +- LocalTableScan [value#176]
      ,None), Statistics(sizeInBytes=12.0 B, hints=none)
      java.lang.AssertionError: assertion failed: InMemoryRelation fields: output, cacheBuilder, statsOfPlanToCache, outputOrdering, values: List(count#178), CachedRDDBuilder(true,10000,StorageLevel(disk, memory, deserialized, 1 replicas),*(1) Project [value#176 AS count#178]
      +- LocalTableScan [value#176]
      ,None), Statistics(sizeInBytes=12.0 B, hints=none)
      	at scala.Predef$.assert(Predef.scala:170)
      	at org.apache.spark.sql.catalyst.trees.TreeNode.jsonFields(TreeNode.scala:611)
      	at org.apache.spark.sql.catalyst.trees.TreeNode.org$apache$spark$sql$catalyst$trees$TreeNode$$collectJsonValue$1(TreeNode.scala:599)
      	at org.apache.spark.sql.catalyst.trees.TreeNode.jsonValue(TreeNode.scala:604)
      	at org.apache.spark.sql.catalyst.trees.TreeNode.toJSON(TreeNode.scala:590)
      

      Attachments

        Activity

          People

            smilegator Xiao Li
            smilegator Xiao Li
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: