Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21791

ORC should support column names with dot

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Duplicate
    • 2.0.0, 2.1.0, 2.2.0
    • 2.3.0
    • SQL
    • None

    Description

      PARQUET

      scala> Seq(Some(1), None).toDF("col.dots").write.parquet("/tmp/parquet_dot")
scala> spark.read.parquet("/tmp/parquet_dot").show
+--------+
|col.dots|
+--------+
|       1|
|    null|
+--------+

      ORC

      scala> Seq(Some(1), None).toDF("col.dots").write.orc("/tmp/orc_dot")
scala> spark.read.orc("/tmp/orc_dot").show
org.apache.spark.sql.catalyst.parser.ParseException:
mismatched input '.' expecting ':'(line 1, pos 10)

== SQL ==
struct<col.dots:int>
----------^^^

      Attachments

        Issue Links

          blocks

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-20901 Feature parity for ORC with Parquet

          • Major - Major loss of function.
          • Open
          is related to

          Question - A formal question. Initially added for the Legal JIRA. SPARK-35274 old hive table's all columns are read when column pruning applies in spark3.0

          • Major - Major loss of function.
          • Open
          is superceded by

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-20682 Add new ORCFileFormat based on Apache ORC

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #19651 [Github] Pull Request #19651 (dongjoon-hyun)

          Web Link PR

          Activity

            People

              Unassigned Unassigned
              dongjoon Dongjoon Hyun
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: