Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23340

Upgrade Apache ORC to 1.4.3

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.3.0
    • 2.3.1, 2.4.0
    • Build, SQL
    • None

    Description

      This issue updates Apache ORC dependencies to 1.4.3 released on February 9th.
      Apache ORC 1.4.2 release removes unnecessary dependencies and 1.4.3 has 5 more patches including bug fixes (https://s.apache.org/Fll8).
      Especially, the following ORC-285 is fixed at 1.4.3.

      scala> val df = Seq(Array.empty[Float]).toDF()

scala> df.write.format("orc").save("/tmp/floatarray")

scala> spark.read.orc("/tmp/floatarray")
res1: org.apache.spark.sql.DataFrame = [value: array<float>]

scala> spark.read.orc("/tmp/floatarray").show()
18/02/12 22:09:10 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 1)
java.io.IOException: Error reading file: file:/tmp/floatarray/part-00000-9c0b461b-4df1-4c23-aac1-3e4f349ac7d6-c000.snappy.orc
	at org.apache.orc.impl.RecordReaderImpl.nextBatch(RecordReaderImpl.java:1191)
	at org.apache.orc.mapreduce.OrcMapreduceRecordReader.ensureBatch(OrcMapreduceRecordReader.java:78)
...
Caused by: java.io.EOFException: Read past EOF for compressed stream Stream for column 2 kind DATA position: 0 length: 0 range: 0 offset: 0 limit: 0

      Attachments

        Issue Links

          blocks

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-20901 Feature parity for ORC with Parquet

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-21783 Turn on ORC filter push-down by default

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Task - A task that needs to be done. SPARK-23426 Use `hive` ORC impl and disable PPD for Spark 2.3.0

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. ORC-285 Empty vector batches of floats or doubles get java.io.EOFException

          • Major - Major loss of function.
          • Closed
          links to

          Pull request #20511 [Github] Pull Request #20511 (dongjoon-hyun)

          Pull request #21093 [Github] Pull Request #21093 (dongjoon-hyun)

          Activity

            People

              dongjoon Dongjoon Hyun
              dongjoon Dongjoon Hyun
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: