Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3200

Replace BufferedBlockMgr with new buffer pool

Agile BoardAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      We want to replace BufferedBlockMgr, a query-wide buffer pool, with a new BufferPool that is shared between all queries. The goals are:

      • Support for guaranteed reservations: i.e. if a reservation is granted, the buffer pool will fulfil it (unless the OS is unable to fulfill the buffer pool's memory requirements).
      • Simplified interaction between reservations and pins (if you have a reservation, you can pin, if you don't, you can't)
      • Support for increasing reservations (up to a planner-specified limit).
      • Support for smaller buffer sizes with similar performance (so we can reduce minimum memory requirement to execute spill-to-disk algorithm)
      • Support for larger buffers to support wide rows
      • Reduced reliance on TCMalloc, which isn't suited to management of large buffers (e.g. see IMPALA-2800)
      • Better transfer model for buffer pool pages, so we can implement transfer-of-ownership consistently for row batches (instead of mixing transfer with MarkNeedsToReturn()).

      Attachments

        Issue Links

        blocks

        Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. IMPALA-4834 All operators should operate within a memory constraint

        • Major - Major loss of function.
        • Open

        Bug - A problem which impairs or prevents the functions of the product. IMPALA-2800 impalad process blocked while releasing memory after a big query

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. IMPALA-2285 Query should return error not cancelled when failing to flush blocks due to lack of disk space

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. IMPALA-4179 Remove RowBatch::MarkNeedsDeepCopy() memory management API

        • Major - Major loss of function.
        • Open

        Improvement - An improvement or enhancement to an existing feature or task. IMPALA-2795 Report spilling time per operator

        • Major - Major loss of function.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. IMPALA-2079 Don't fail when write to scratch dir results in error.

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. IMPALA-3706 Planner should set min/max per-node buffer pool reservations to optimize query performance

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved

        Task - A task that needs to be done. IMPALA-3265 Create a metrics to track spilling per operator

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved
        is depended upon by

        Improvement - An improvement or enhancement to an existing feature or task. IMPALA-5165 Allocate memory for all data from BufferPool

        • Major - Major loss of function.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. IMPALA-1382 Wasted space in buffered-tuple-stream in presence of many NULL tuples

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. IMPALA-2167 Remove the old (unpartitioned) HJ and AGG nodes

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved

        Task - A task that needs to be done. IMPALA-5656 Update BlockMgr.BytesWritten reference in docs for new buffer pool

        • Major - Major loss of function.
        • Resolved

        Task - A task that needs to be done. IMPALA-5655 Update docs for max_block_mgr_memory/buffer_pool_limit

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Closed
        is duplicated by

        Improvement - An improvement or enhancement to an existing feature or task. IMPALA-5701 Additional mem_limit settings for resource management (min mem, max before spill, max spill)

        • Major - Major loss of function.
        • Closed
        is related to

        Sub-task - The sub-task of the issue IMPALA-5161 TPC-DS Q78 with MEM_LIMIT=10GB fails with "Repartitioning did not reduce the size of a spilled partition" on BufferPool dev branch

        • Critical - Crashes, loss of data, severe memory leak.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. IMPALA-5778 Clarify logging around and usage of read_size startup option

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. IMPALA-2285 Query should return error not cancelled when failing to flush blocks due to lack of disk space

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. IMPALA-5169 Parallelise read I/O of BufferPool::Pin()

        • Major - Major loss of function.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. IMPALA-5229 Try using TCMalloc + Huge Pages for buffers

        • Major - Major loss of function.
        • Resolved

        Documentation - Documentation or Website IMPALA-5786 Document buffer pool changes

        • Blocker - Blocks development and/or testing work, production could not run
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. IMPALA-4981 COMPUTE STATS with MT_DOP=1 and tight memory limit produces spilling error

        • Blocker - Blocks development and/or testing work, production could not run
        • Resolved
        relates to

        New Feature - A new feature of the product, which has yet to be developed. IMPALA-3700 Introduce upper limit to hash table size or HashBuckets for Streaming agg

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved
        1.
        		 Implement basic in-memory buffer pool Sub-task Resolved Tim Armstrong Actions
        2.
        		 Add spilling support to new buffer pool Sub-task Resolved Tim Armstrong Actions
        3.
        		 Integrate buffer pool reservations with memtrackers Sub-task Resolved Tim Armstrong Actions
        4.
        		 Set up query mem tracker in QueryState Sub-task Resolved Tim Armstrong Actions
        5.
        		 Clients can violate BufferPool invariants by calling ReservationTracker methods directly. Sub-task Resolved Tim Armstrong Actions
        6.
        		 Buffer pool unpinned invariant does not take into account multiply-pinned bytes Sub-task Resolved Tim Armstrong Actions
        7.
        		 MemTracker::EnableReservationReporting() is not thread-safe Sub-task Resolved Tim Armstrong Actions
        8.
        		 Compute memory reservation in planner and claim atomically in Prepare() Sub-task Resolved Tim Armstrong Actions
        9.
        		 Port spilling ExecNodes to new buffer pool Sub-task Resolved Tim Armstrong Actions
        10.
        		 Implement scalable buffer recycling in buffer pool Sub-task Resolved Tim Armstrong Actions
        11.
        		 Backend support for large rows Sub-task Resolved Tim Armstrong Actions
        12.
        		 Backend support for large rows in BufferedTupleStream Sub-task Resolved Tim Armstrong Actions
        13.
        		 Fix BufferPool handling of scratch read errors Sub-task Resolved Tim Armstrong Actions
        14.
        		 Port relevant BufferedBlockMgr unit tests for BufferPool Sub-task Resolved Tim Armstrong Actions
        15.
        		 Add reservation stress option for test coverage Sub-task Resolved Tim Armstrong Actions
        16.
        		 Validate and fix spilling performance and memory usage of new buffer pool Sub-task Resolved Tim Armstrong Actions
        17.
        		 Queries with a large number of small joins regress in terms of memory usage due to memory reservation Sub-task Resolved Tim Armstrong Actions
        18.
        		 TPC-DS Q78 with MEM_LIMIT=10GB fails with "Repartitioning did not reduce the size of a spilled partition" on BufferPool dev branch Sub-task Resolved Tim Armstrong Actions
        19.
        		 Account for difference between process memory consumption and memory used by queries Sub-task Resolved Tim Armstrong Actions
        20.
        		 Clean up BufferPool profile counters Sub-task Resolved Tim Armstrong Actions
        21.
        		 Consider reducing partition fanout in Hash Join Sub-task Resolved Tim Armstrong Actions
        22.
        		 Partitioned aggregation node repartitions when spilled partition could fit in memory Sub-task Resolved Tim Armstrong Actions
        23.
        		 Ensure that NAAJ works with spilling enabled and disabled. Sub-task Resolved Tim Armstrong Actions
        24.
        		 Performance regresses on buffer pool dev branch for high-ndv aggregations Sub-task Resolved Tim Armstrong Actions
        25.
        		 list::size() in BufferedTupleStreamV2::AdvanceWritePage() is expensive Sub-task Resolved Tim Armstrong Actions
        26.
        		 Consider allowing configuration of buffer pool size Sub-task Resolved Tim Armstrong Actions
        27.
        		 Track exchange node buffers memory as part of memory reservation Sub-task Resolved Tim Armstrong Actions
        28.
        		 BufferedTupleStreamV2::CheckConsistency() is too slow for large streams with small pages in Debug build Sub-task Resolved Tim Armstrong Actions
        29.
        		 Consider ways to reduce the accumulation of clean pages when executing large spilling queries Sub-task Resolved Tim Armstrong Actions
        30.
        		 Eagerly release reservation in blocking nodes Sub-task Resolved Tim Armstrong Actions
        31.
        		 Consider always reserving memory for grouping pre-aggregations Sub-task Resolved Tim Armstrong Actions
        32.
        		 Ensure test coverage for spilling disabled for all spilling operators Sub-task Resolved Tim Armstrong Actions
        33.
        		 Consider reducing RESERVATION_MIN_MEM_REMAINING Sub-task Resolved Tim Armstrong Actions
        34.
        		 SET_DENY_RESERVATION_PROBABILITY debug action is not always effective Sub-task Resolved Tim Armstrong Actions

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            tarmstrong Tim Armstrong
            tarmstrong Tim Armstrong
            Votes:
            1 Vote for this issue
            Watchers:
            17 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment