Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3200

Replace BufferedBlockMgr with new buffer pool

    XMLWordPrintableJSON

Details

    Description

      We want to replace BufferedBlockMgr, a query-wide buffer pool, with a new BufferPool that is shared between all queries. The goals are:

      • Support for guaranteed reservations: i.e. if a reservation is granted, the buffer pool will fulfil it (unless the OS is unable to fulfill the buffer pool's memory requirements).
      • Simplified interaction between reservations and pins (if you have a reservation, you can pin, if you don't, you can't)
      • Support for increasing reservations (up to a planner-specified limit).
      • Support for smaller buffer sizes with similar performance (so we can reduce minimum memory requirement to execute spill-to-disk algorithm)
      • Support for larger buffers to support wide rows
      • Reduced reliance on TCMalloc, which isn't suited to management of large buffers (e.g. see IMPALA-2800)
      • Better transfer model for buffer pool pages, so we can implement transfer-of-ownership consistently for row batches (instead of mixing transfer with MarkNeedsToReturn()).

      Attachments

        Issue Links

          blocks

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. IMPALA-4834 All operators should operate within a memory constraint

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-2800 impalad process blocked while releasing memory after a big query

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-2285 Query should return error not cancelled when failing to flush blocks due to lack of disk space

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-4179 Remove RowBatch::MarkNeedsDeepCopy() memory management API

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-2795 Report spilling time per operator

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-2079 Don't fail when write to scratch dir results in error.

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-3706 Planner should set min/max per-node buffer pool reservations to optimize query performance

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Task - A task that needs to be done. IMPALA-3265 Create a metrics to track spilling per operator

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is depended upon by

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-5165 Allocate memory for all data from BufferPool

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-1382 Wasted space in buffered-tuple-stream in presence of many NULL tuples

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-2167 Remove the old (unpartitioned) HJ and AGG nodes

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Task - A task that needs to be done. IMPALA-5656 Update BlockMgr.BytesWritten reference in docs for new buffer pool

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. IMPALA-5655 Update docs for max_block_mgr_memory/buffer_pool_limit

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-5701 Additional mem_limit settings for resource management (min mem, max before spill, max spill)

          • Major - Major loss of function.
          • Closed
          is related to

          Sub-task - The sub-task of the issue IMPALA-5161 TPC-DS Q78 with MEM_LIMIT=10GB fails with "Repartitioning did not reduce the size of a spilled partition" on BufferPool dev branch

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-5778 Clarify logging around and usage of read_size startup option

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-2285 Query should return error not cancelled when failing to flush blocks due to lack of disk space

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-5169 Parallelise read I/O of BufferPool::Pin()

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-5229 Try using TCMalloc + Huge Pages for buffers

          • Major - Major loss of function.
          • Resolved

          Documentation - Documentation or Website IMPALA-5786 Document buffer pool changes

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-4981 COMPUTE STATS with MT_DOP=1 and tight memory limit produces spilling error

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          relates to

          New Feature - A new feature of the product, which has yet to be developed. IMPALA-3700 Introduce upper limit to hash table size or HashBuckets for Streaming agg

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          There are no Sub-Tasks for this issue.

          Activity

            People

              tarmstrong Tim Armstrong
              tarmstrong Tim Armstrong
              Votes:
              1 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: