Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3200 Replace BufferedBlockMgr with new buffer pool
  3. IMPALA-3205

Validate and fix spilling performance and memory usage of new buffer pool

    XMLWordPrintableJSON

Details

    Description

      We want to ensure that performance doesn't regress:

      • Memory usage for the new buffer pool (with the old small buffers optimisation removed) should be similar or less than the old code for typical workloads. This will require reducing the buffer size - e.g. to 1MB or less.
      • Spill-to-disk performance should not regress more than is acceptable. This may be tricky to get right since the I/O size will be reduced from 8MB. We may need to allocate logically sequential blocks in contiguous disk and do some kind of prefetching in the buffer pool to issue multiple writes at a time.

      Attachments

        1. min_mem_old_aggs_joins.csv
          0.2 kB
          Tim Armstrong
        2. min_mem.py
          0.5 kB
          Tim Armstrong
        3. min_mem_buffer_pool_default.csv
          0.2 kB
          Tim Armstrong
        4. min_mem_part_aggs_joins.csv
          0.2 kB
          Tim Armstrong
        5. min_mem_buffer_pool_64kb_buffers.csv
          0.2 kB
          Tim Armstrong
        6. spilling-agg-bbm.txt
          27 kB
          Tim Armstrong
        7. spilling-agg-bufferpool.txt
          55 kB
          Tim Armstrong
        8. spilling-agg-bufferpool-8mb-buffers.txt
          55 kB
          Tim Armstrong
        9. spilling-sort-bbm.txt
          2 kB
          Tim Armstrong
        10. spilling-sort-bufferpool.txt
          4 kB
          Tim Armstrong
        11. min_mem_tpcds_5_bp.csv
          0.3 kB
          Tim Armstrong
        12. min_mem_tpcds_5_bbm.csv
          0.3 kB
          Tim Armstrong
        13. min_mem_tpcds_5_bp.json
          55 kB
          Tim Armstrong
        14. min_mem_tpcds_5_bbm.json
          55 kB
          Tim Armstrong
        15. min_mem_tpcds_5_bp_reduce_min_remaining.csv
          0.3 kB
          Tim Armstrong
        16. min_mem_tpcds_5_bp_reduce_min_remaining.json
          55 kB
          Tim Armstrong
        17. spilling-hj2-bp.txt
          56 kB
          Tim Armstrong
        18. spilling-hj2-bbm.txt
          98 kB
          Tim Armstrong
        19. spilling-hj2-bp-8mb-buffers.txt
          33 kB
          Tim Armstrong

        Activity

          People

            tarmstrong Tim Armstrong
            tarmstrong Tim Armstrong
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: