Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26709

OptimizeMetadataOnlyQuery does not correctly handle the files with zero record

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 2.1.0, 2.1.3, 2.2.3, 2.3.2, 2.4.0
    • 2.3.3, 2.4.1, 3.0.0
    • SQL

    Description

      import org.apache.spark.sql.functions.lit
withSQLConf(SQLConf.OPTIMIZER_METADATA_ONLY.key -> "true") {
  withTempPath { path =>
    val tabLocation = path.getAbsolutePath
    val partLocation = new Path(path.getAbsolutePath, "partCol1=3")
    val df = spark.emptyDataFrame.select(lit(1).as("col1"))
    df.write.parquet(partLocation.toString)
    val readDF = spark.read.parquet(tabLocation)
    checkAnswer(readDF.selectExpr("max(partCol1)"), Row(null))
    checkAnswer(readDF.selectExpr("max(col1)"), Row(null))
  }
}

      OptimizeMetadataOnlyQuery has a correctness bug to handle the file with the empty records for partitioned tables. The above test will fail in 2.4, which can generate an empty file, but the underlying issue in the read path still exists in 2.3, 2.2 and 2.1.

      Attachments

        Issue Links

          is caused by

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-15752 Optimize metadata only query that has an aggregate whose children are deterministic project or filter operators

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. SPARK-26996 Scalar Subquery not handled properly in Spark 2.4

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-34194 Queries that only touch partition columns shouldn't scan through all files

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          links to

          Pull request #23648 [Github] Pull Request #23648 (gengliangwang)

          Pull request #23648 [Github] Pull Request #23648 (gengliangwang)

          Web Link GitHub Pull Request #23635

          Web Link GitHub Pull Request #23648

          Activity

            People

              Gengliang.Wang Gengliang Wang
              smilegator Xiao Li
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: