Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-13985

ORC improvements for reducing the file system calls in task side

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.3.0, 2.2.0
    • 1.3.0, 2.1.1, 2.2.0
    • ORC
    • None

    Description

      HIVE-13840 fixed some issues with addition file system invocations during split generation. Similarly, this jira will fix issues with additional file system invocations on the task side. To avoid reading footers on the task side, users can set hive.orc.splits.include.file.footer to true which will serialize the orc footers on the splits. But this has issues with serializing unwanted information like column statistics and other metadata which are not really required for reading orc split on the task side. We can reduce the payload on the orc splits by serializing only the minimum required information (stripe information, types, compression details). This will decrease the payload on the orc splits and can potentially avoid OOMs in application master (AM) during split generation. This jira also address other issues concerning the AM cache. The local cache used by AM is soft reference cache. This can introduce unpredictability across multiple runs of the same query. We can cache the serialized footer in the local cache and also use strong reference cache which should avoid memory pressure and will have better predictability.

      One other improvement that we can do is when hive.orc.splits.include.file.footer is set to false, on the task side we make one additional file system call to know the size of the file. If we can serialize the file length in the orc split this can be avoided.

      Attachments

        1. HIVE-13985.1.patch
          150 kB
          Prasanth Jayachandran
        2. HIVE-13985.2.patch
          147 kB
          Prasanth Jayachandran
        3. HIVE-13985-branch-2.1.patch
          147 kB
          Prasanth Jayachandran
        4. HIVE-13985-branch-1.patch
          123 kB
          Prasanth Jayachandran
        5. HIVE-13985-branch-1.patch
          123 kB
          Prasanth Jayachandran
        6. HIVE-13985-branch-1.patch
          126 kB
          Prasanth Jayachandran
        7. HIVE-13985-branch-1.patch
          126 kB
          Prasanth Jayachandran
        8. HIVE-13985.3.patch
          166 kB
          Prasanth Jayachandran
        9. HIVE-13985.4.patch
          167 kB
          Prasanth Jayachandran
        10. HIVE-13985.5.patch
          167 kB
          Prasanth Jayachandran
        11. HIVE-13985.6.patch
          167 kB
          Prasanth Jayachandran

        Issue Links

          Activity

            People

              prasanth_j Prasanth Jayachandran
              prasanth_j Prasanth Jayachandran
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: