Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-15665

LLAP: OrcFileMetadata objects in cache can impact heap usage

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 3.0.0
    • llap

    Description

      OrcFileMetadata internally has filestats, stripestats etc which are allocated in heap. On large data sets, this could have an impact on the heap usage and the memory usage by different executors in LLAP.

      Attachments

        1. HIVE-15665.01.patch
          331 kB
          Sergey Shelukhin
        2. HIVE-15665.02.patch
          331 kB
          Sergey Shelukhin
        3. HIVE-15665.03.patch
          142 kB
          Sergey Shelukhin
        4. HIVE-15665.04.patch
          142 kB
          Sergey Shelukhin
        5. HIVE-15665.05.patch
          140 kB
          Sergey Shelukhin
        6. HIVE-15665.06.patch
          140 kB
          Sergey Shelukhin
        7. HIVE-15665.07.patch
          140 kB
          Sergey Shelukhin
        8. HIVE-15665.08.patch
          140 kB
          Sergey Shelukhin
        9. HIVE-15665.09.patch
          156 kB
          Sergey Shelukhin
        10. HIVE-15665.10.patch
          158 kB
          Sergey Shelukhin
        11. HIVE-15665.11.patch
          158 kB
          Sergey Shelukhin
        12. HIVE-15665.12.patch
          158 kB
          Sergey Shelukhin
        13. HIVE-15665.patch
          274 kB
          Sergey Shelukhin

        Issue Links

        blocks

        Bug - A problem which impairs or prevents the functions of the product. HIVE-17524 instrument LLAP metadata cache with separate counters by format

        • Major - Major loss of function.
        • Open
        is blocked by

        Bug - A problem which impairs or prevents the functions of the product. HIVE-16233 llap: Query failed with AllocatorOutOfMemoryException

        • Major - Major loss of function.
        • Closed
        is related to

        Bug - A problem which impairs or prevents the functions of the product. HIVE-16224 modify off-heap ORC metadata cache after ORC changes

        • Major - Major loss of function.
        • Open
        relates to

        Bug - A problem which impairs or prevents the functions of the product. ORC-159 expose metadata buffer from metadata reader (for caching)

        • Major - Major loss of function.
        • Open

        Bug - A problem which impairs or prevents the functions of the product. ORC-158 do not assume an array-backed buffer in static ReaderImpl methods

        • Major - Major loss of function.
        • In Progress

        Bug - A problem which impairs or prevents the functions of the product. HIVE-16318 LLAP cache: address some issues in 2.2/2.3

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. ORC-160 expose index read planning in RecordReaderUtils

        • Major - Major loss of function.
        • Closed
        links to

        Web Link RB

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            sershe Sergey Shelukhin Assign to me
            rajesh.balamohan Rajesh Balamohan
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment