Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-20682

Add new ORCFileFormat based on Apache ORC

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.4.1, 1.5.2, 1.6.3, 2.1.1, 2.2.0
    • 2.3.0
    • SQL

    Description

      Since SPARK-2883, Apache Spark supports Apache ORC inside `sql/hive` module with Hive dependency. This issue aims to add a new and faster ORC data source inside `sql/core` and to replace the old ORC data source eventually. In this issue, the latest Apache ORC 1.4.0 (released yesterday) is used.

      There are four key benefits.

      • Speed: Use both Spark `ColumnarBatch` and ORC `RowBatch` together. This is faster than the current implementation in Spark.
      • Stability: Apache ORC 1.4.0 has many fixes and we can depend on ORC community more.
      • Usability: User can use `ORC` data sources without hive module, i.e, `-Phive`.
      • Maintainability: Reduce the Hive dependency and can remove old legacy code later.

      Attachments

        Issue Links

          blocks

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-20901 Feature parity for ORC with Parquet

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-20728 Make ORCFileFormat configurable between sql/hive and sql/core

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-21787 Support for pushing down filters for DateType in native OrcFileFormat

          • Major - Major loss of function.
          • Resolved
          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. SPARK-21422 Depend on Apache ORC 1.4.0

          • Major - Major loss of function.
          • Resolved
          is related to

          Question - A formal question. Initially added for the Legal JIRA. SPARK-35274 old hive table's all columns are read when column pruning applies in spark3.0

          • Major - Major loss of function.
          • Open
          supercedes

          Bug - A problem which impairs or prevents the functions of the product. SPARK-19109 ORC metadata section can sometimes exceed protobuf message size limit

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-21791 ORC should support column names with dot

          • Major - Major loss of function.
          • Closed
          links to

          Pull request #17924 [Github] Pull Request #17924 (dongjoon-hyun)

          Pull request #17943 [Github] Pull Request #17943 (dongjoon-hyun)

          Pull request #18953 [Github] Pull Request #18953 (dongjoon-hyun)

          Pull request #18953 [Github] Pull Request #18953 (dongjoon-hyun)

          Pull request #19651 [Github] Pull Request #19651 (dongjoon-hyun)

          Activity

            People

              dongjoon Dongjoon Hyun
              dongjoon Dongjoon Hyun
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: