Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33507 Improve and fix cache behavior in v1 and v2
  3. SPARK-33963

`isCached` return `false` for cached Hive table

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.0, 3.2.0
    • 3.0.2, 3.1.0
    • SQL
    • None

    Description

      The same works in Spark 3.0 but fails in Spark 3.2.0-SNAPSHOT (and 3.1.0 probably):
      Spark 3.0:

      Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.0.1
      /_/
scala> sql("CREATE TABLE tbl (col int)")
res2: org.apache.spark.sql.DataFrame = []
scala> spark.catalog.isCached("tbl")
res3: Boolean = false

scala> sql("CACHE TABLE tbl")
res4: org.apache.spark.sql.DataFrame = []

scala> spark.catalog.isCached("tbl")
res5: Boolean = true

      Spark master:

      Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.2.0-SNAPSHOT
      /_/
scala> sql("CREATE TABLE tbl (col int)")
res1: org.apache.spark.sql.DataFrame = []

scala> spark.catalog.isCached("tbl")
res2: Boolean = false

scala> sql("CACHE TABLE tbl")
res3: org.apache.spark.sql.DataFrame = []

scala> spark.catalog.isCached("tbl")
res4: Boolean = false

      Attachments

        Activity

          People

            maxgekk Max Gekk
            maxgekk Max Gekk
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: