Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28208

Upgrade to ORC 1.5.6

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • Build, SQL
    • None

    Description

      As part of the ORC 1.5.6 release, we optimized the common pattern of:

      Reader reader = OrcFile.createReader(...);
      RecordReader rows = reader.rows(...);

      which used to open one file handle in the Reader and a second one in the RecordReader. Users were seeing this as a regression when moving from the old Spark ORC reader via hive to the new native reader, because it opened twice as many files on the NameNode.

      In ORC 1.5.6, we changed the ORC library so that it keeps the file handle in the Reader until it is either closed or a RecordReader is created from it. This has cut down the number of file open requests on the NameNode by half in typical spark applications. (Hive's ORC code avoided this problem by putting the file footer in to the input splits, but that has other problems.)

      To get the new optimization without leaking file handles, Spark needs to be close the readers that aren't used to create RecordReaders.

      Attachments

        Issue Links

          Activity

            People

              omalley Owen O'Malley
              omalley Owen O'Malley
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: