[IMPALA-9712] Hit OOM on TPC-H Q19 - ASF JIRA

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: Impala 4.0.0
Component/s: Backend
Labels:
- multithreading
- resource-management

Epic Color:
ghx-label-10

Description

We saw an out-of-memory on TPC-H 1TB with mt_dop=16.

    Query Status: Memory limit exceeded: ParquetColumnChunkReader::ReadDataPage() failed to allocate 65454 bytes for decompressed data.
HDFS_SCAN_NODE (id=0) could not allocate 63.92 KB without exceeding limit.
Error occurred on backend impala-executor-000-1:22000 by fragment 4b497d5e4237002a:94c50a5200000041
Memory left in process limit: 71.74 GB
Memory left in query limit: 52.96 KB
Query(4b497d5e4237002a:94c50a5200000000): Limit=2.35 GB Reservation=1.56 GB ReservationLimit=1.88 GB OtherMemory=801.69 MB Total=2.35 GB Peak=2.35 GB
  Runtime Filter Bank: Reservation=1.00 MB ReservationLimit=1.00 MB OtherMemory=0 Total=1.00 MB Peak=1.00 MB
  Fragment 4b497d5e4237002a:94c50a52000000a7: Reservation=0 OtherMemory=0 Total=0 Peak=51.91 MB
    HDFS_SCAN_NODE (id=1): Reservation=0 OtherMemory=0 Total=0 Peak=48.78 MB
    KrpcDataStreamSender (dst_id=4): Total=0 Peak=31.17 KB
  CodeGen: Total=7.82 KB Peak=2.52 MB
  Fragment 4b497d5e4237002a:94c50a52000000a8: Reservation=0 OtherMemory=0 Total=0 Peak=40.15 MB
    HDFS_SCAN_NODE (id=1): Reservation=0 OtherMemory=0 Total=0 Peak=36.84 MB
    KrpcDataStreamSender (dst_id=4): Total=0 Peak=31.17 KB
  Fragment 4b497d5e4237002a:94c50a52000000a9: Reservation=0 OtherMemory=0 Total=0 Peak=31.79 MB
    HDFS_SCAN_NODE (id=1): Reservation=0 OtherMemory=0 Total=0 Peak=28.78 MB
    KrpcDataStreamSender (dst_id=4): Total=0 Peak=31.17 KB
  Fragment 4b497d5e4237002a:94c50a52000000c3: Reservation=544.00 MB OtherMemory=21.12 KB Total=544.02 MB Peak=544.36 MB
    EXCHANGE_NODE (id=4): Reservation=0 OtherMemory=0 Total=0 Peak=352.00 KB
      KrpcDeferredRpcs: Total=0 Peak=0
    Hash Join Builder (join_node_id=2): Reservation=544.00 MB OtherMemory=13.12 KB Total=544.01 MB Peak=544.02 MB
      Hash Join Builder (join_node_id=2) Exprs: Total=13.12 KB Peak=13.12 KB
  Fragment 4b497d5e4237002a:94c50a5200000041: Reservation=66.00 MB OtherMemory=51.15 MB Total=117.15 MB Peak=117.29 MB
    AGGREGATION_NODE (id=3): Total=43.83 MB Peak=49.86 MB
      NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
        Exprs: Total=4.00 KB Peak=4.00 KB
    HASH_JOIN_NODE (id=2): Reservation=0 OtherMemory=6.04 MB Total=6.04 MB Peak=8.29 MB
      Exprs: Total=41.12 KB Peak=41.12 KB
    HDFS_SCAN_NODE (id=0): Reservation=66.00 MB OtherMemory=1.26 MB Total=67.26 MB Peak=67.42 MB
      Exprs: Total=8.00 KB Peak=8.00 KB
    KrpcDataStreamSender (dst_id=5): Total=14.22 KB Peak=14.22 KB
  Fragment 4b497d5e4237002a:94c50a5200000042: Reservation=66.00 MB OtherMemory=52.06 MB Total=118.06 MB Peak=118.06 MB
    AGGREGATION_NODE (id=3): Total=50.63 MB Peak=50.63 MB
      NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
        Exprs: Total=4.00 KB Peak=4.00 KB
    HASH_JOIN_NODE (id=2): Reservation=0 OtherMemory=57.12 KB Total=57.12 KB Peak=7.79 MB
      Exprs: Total=41.12 KB Peak=41.12 KB
    HDFS_SCAN_NODE (id=0): Reservation=66.00 MB OtherMemory=1.35 MB Total=67.35 MB Peak=67.42 MB
      Exprs: Total=8.00 KB Peak=8.00 KB
    KrpcDataStreamSender (dst_id=5): Total=14.22 KB Peak=14.22 KB
  CodeGen: Total=4.06 KB Peak=778.50 KB
  Fragment 4b497d5e4237002a:94c50a5200000043: Reservation=66.00 MB OtherMemory=47.81 MB Total=113.81 MB Peak=116.01 MB
    AGGREGATION_NODE (id=3): Total=45.01 MB Peak=48.67 MB
      NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
        Exprs: Total=4.00 KB Peak=4.00 KB
    HASH_JOIN_NODE (id=2): Reservation=0 OtherMemory=1.48 MB Total=1.48 MB Peak=7.79 MB
      Exprs: Total=41.12 KB Peak=41.12 KB
    HDFS_SCAN_NODE (id=0): Reservation=66.00 MB OtherMemory=1.35 MB Total=67.35 MB Peak=67.42 MB
      Exprs: Total=8.00 KB Peak=8.00 KB
    KrpcDataStreamSender (dst_id=5): Total=14.22 KB Peak=14.22 KB
  Fragment 4b497d5e4237002a:94c50a5200000044: Reservation=66.00 MB OtherMemory=56.25 MB Total=122.25 MB Peak=122.25 MB
    AGGREGATION_NODE (id=3): Total=54.81 MB Peak=54.81 MB
      NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
        Exprs: Total=4.00 KB Peak=4.00 KB
    HASH_JOIN_NODE (id=2): Reservation=0 OtherMemory=57.12 KB Total=57.12 KB Peak=8.23 MB
      Exprs: Total=41.12 KB Peak=41.12 KB
    HDFS_SCAN_NODE (id=0): Reservation=66.00 MB OtherMemory=1.35 MB Total=67.35 MB Peak=67.42 MB
      Exprs: Total=8.00 KB Peak=8.00 KB
    KrpcDataStreamSender (dst_id=5): Total=14.22 KB Peak=14.22 KB
  Fragment 4b497d5e4237002a:94c50a5200000045: Reservation=66.00 MB OtherMemory=49.68 MB Total=115.68 MB Peak=115.68 MB
    AGGREGATION_NODE (id=3): Total=43.95 MB Peak=44.61 MB
      NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
        Exprs: Total=4.00 KB Peak=4.00 KB
    HASH_JOIN_NODE (id=2): Reservation=0 OtherMemory=4.36 MB Total=4.36 MB Peak=8.29 MB
      Exprs: Total=41.12 KB Peak=41.12 KB
    HDFS_SCAN_NODE (id=0): Reservation=66.00 MB OtherMemory=1.35 MB Total=67.35 MB Peak=67.42 MB
      Exprs: Total=8.00 KB Peak=8.00 KB
    KrpcDataStreamSender (dst_id=5): Total=14.22 KB Peak=14.22 KB
  Fragment 4b497d5e4237002a:94c50a5200000046: Reservation=66.00 MB OtherMemory=48.62 MB Total=114.62 MB Peak=115.70 MB
    AGGREGATION_NODE (id=3): Total=42.33 MB Peak=48.36 MB
      NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
        Exprs: Total=4.00 KB Peak=4.00 KB
    HASH_JOIN_NODE (id=2): Reservation=0 OtherMemory=4.92 MB Total=4.92 MB Peak=8.86 MB
      Exprs: Total=41.12 KB Peak=41.12 KB
    HDFS_SCAN_NODE (id=0): Reservation=66.00 MB OtherMemory=1.35 MB Total=67.35 MB Peak=67.42 MB
      Exprs: Total=8.00 KB Peak=8.00 KB
    KrpcDataStreamSender (dst_id=5): Total=14.22 KB Peak=14.22 KB
  Fragment 4b497d5e4237002a:94c50a5200000047: Reservation=66.00 MB OtherMemory=49.53 MB Total=115.53 MB Peak=115.53 MB
    AGGREGATION_NODE (id=3): Total=44.45 MB Peak=47.74 MB
      NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
        Exprs: Total=4.00 KB Peak=4.00 KB
    HASH_JOIN_NODE (id=2): Reservation=0 OtherMemory=3.79 MB Total=3.79 MB Peak=8.23 MB
      Exprs: Total=41.12 KB Peak=41.12 KB
    HDFS_SCAN_NODE (id=0): Reservation=66.00 MB OtherMemory=1.26 MB Total=67.26 MB Peak=67.42 MB
      Exprs: Total=8.00 KB Peak=8.00 KB
    KrpcDataStreamSender (dst_id=5): Total=14.22 KB Peak=14.22 KB
  Fragment 4b497d5e4237002a:94c50a5200000048: Reservation=66.00 MB OtherMemory=54.38 MB Total=120.38 MB Peak=120.38 MB
    AGGREGATION_NODE (id=3): Total=53.13 MB Peak=53.13 MB
      NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
        Exprs: Total=4.00 KB Peak=4.00 KB
    HASH_JOIN_NODE (id=2): Reservation=0 OtherMemory=57.12 KB Total=57.12 KB Peak=8.29 MB
      Exprs: Total=41.12 KB Peak=41.12 KB
    HDFS_SCAN_NODE (id=0): Reservation=66.00 MB OtherMemory=1.17 MB Total=67.17 MB Peak=67.42 MB
      Exprs: Total=8.00 KB Peak=8.00 KB
    KrpcDataStreamSender (dst_id=5): Total=14.22 KB Peak=14.22 KB
  Fragment 4b497d5e4237002a:94c50a5200000049: Reservation=66.00 MB OtherMemory=47.81 MB Total=113.81 MB Peak=115.08 MB
    AGGREGATION_NODE (id=3): Total=45.45 MB Peak=47.73 MB
      NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
        Exprs: Total=4.00 KB Peak=4.00 KB
    HASH_JOIN_NODE (id=2): Reservation=0 OtherMemory=1005.84 KB Total=1005.84 KB Peak=8.23 MB
      Exprs: Total=41.12 KB Peak=41.12 KB
    HDFS_SCAN_NODE (id=0): Reservation=66.00 MB OtherMemory=1.35 MB Total=67.35 MB Peak=67.42 MB
      Exprs: Total=8.00 KB Peak=8.00 KB
    KrpcDataStreamSender (dst_id=5): Total=14.22 KB Peak=14.22 KB
  Fragment 4b497d5e4237002a:94c50a520000004a: Reservation=66.00 MB OtherMemory=50.08 MB Total=116.08 MB Peak=117.15 MB
    AGGREGATION_NODE (id=3): Total=44.95 MB Peak=47.73 MB
      NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
        Exprs: Total=4.00 KB Peak=4.00 KB
    HASH_JOIN_NODE (id=2): Reservation=0 OtherMemory=5.10 MB Total=5.10 MB Peak=7.79 MB
      Exprs: Total=41.12 KB Peak=41.12 KB
    HDFS_SCAN_NODE (id=0): Reservation=66.00 MB OtherMemory=4.00 KB Total=66.00 MB Peak=67.42 MB
      Exprs: Total=4.00 KB Peak=8.00 KB
    KrpcDataStreamSender (dst_id=5): Total=14.22 KB Peak=14.22 KB
  CodeGen: Total=53.44 KB Peak=13.23 MB
  Fragment 4b497d5e4237002a:94c50a520000004b: Reservation=66.00 MB OtherMemory=47.81 MB Total=113.81 MB Peak=115.08 MB
    AGGREGATION_NODE (id=3): Total=45.95 MB Peak=47.74 MB
      NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
        Exprs: Total=4.00 KB Peak=4.00 KB
    HASH_JOIN_NODE (id=2): Reservation=0 OtherMemory=685.92 KB Total=685.92 KB Peak=8.29 MB
      Exprs: Total=41.12 KB Peak=41.12 KB
    HDFS_SCAN_NODE (id=0): Reservation=66.00 MB OtherMemory=1.17 MB Total=67.17 MB Peak=67.42 MB
      Exprs: Total=8.00 KB Peak=8.00 KB
    KrpcDataStreamSender (dst_id=5): Total=14.22 KB Peak=14.22 KB
  Fragment 4b497d5e4237002a:94c50a520000004c: Reservation=66.00 MB OtherMemory=48.34 MB Total=114.34 MB Peak=117.29 MB
    AGGREGATION_NODE (id=3): Total=43.83 MB Peak=49.86 MB
      NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
        Exprs: Total=4.00 KB Peak=4.00 KB
    HASH_JOIN_NODE (id=2): Reservation=0 OtherMemory=3.23 MB Total=3.23 MB Peak=8.29 MB
      Exprs: Total=41.12 KB Peak=41.12 KB
    HDFS_SCAN_NODE (id=0): Reservation=66.00 MB OtherMemory=1.26 MB Total=67.26 MB Peak=67.42 MB
      Exprs: Total=8.00 KB Peak=8.00 KB
    KrpcDataStreamSender (dst_id=5): Total=14.22 KB Peak=14.22 KB
  Fragment 4b497d5e4237002a:94c50a520000004d: Reservation=66.00 MB OtherMemory=45.07 MB Total=111.07 MB Peak=115.08 MB
    AGGREGATION_NODE (id=3): Total=37.84 MB Peak=47.73 MB
      NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
        Exprs: Total=4.00 KB Peak=4.00 KB
    HASH_JOIN_NODE (id=2): Reservation=0 OtherMemory=6.04 MB Total=6.04 MB Peak=7.79 MB
      Exprs: Total=41.12 KB Peak=41.12 KB
    HDFS_SCAN_NODE (id=0): Reservation=66.00 MB OtherMemory=1.17 MB Total=67.17 MB Peak=67.42 MB
      Exprs: Total=8.00 KB Peak=8.00 KB
    KrpcDataStreamSender (dst_id=5): Total=14.22 KB Peak=14.22 KB
  Fragment 4b497d5e4237002a:94c50a520000004e: Reservation=66.00 MB OtherMemory=45.81 MB Total=111.81 MB Peak=113.58 MB
    AGGREGATION_NODE (id=3): Total=43.95 MB Peak=46.24 MB
      NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
        Exprs: Total=4.00 KB Peak=4.00 KB
    HASH_JOIN_NODE (id=2): Reservation=0 OtherMemory=685.92 KB Total=685.92 KB Peak=7.73 MB
      Exprs: Total=41.12 KB Peak=41.12 KB
    HDFS_SCAN_NODE (id=0): Reservation=66.00 MB OtherMemory=1.17 MB Total=67.17 MB Peak=67.33 MB
      Exprs: Total=8.00 KB Peak=8.00 KB
    KrpcDataStreamSender (dst_id=5): Total=14.22 KB Peak=14.22 KB
  Fragment 4b497d5e4237002a:94c50a520000004f: Reservation=66.00 MB OtherMemory=54.59 MB Total=120.59 MB Peak=120.59 MB
    AGGREGATION_NODE (id=3): Total=53.25 MB Peak=53.25 MB
      NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
        Exprs: Total=4.00 KB Peak=4.00 KB
    HASH_JOIN_NODE (id=2): Reservation=0 OtherMemory=57.12 KB Total=57.12 KB Peak=8.29 MB
      Exprs: Total=41.12 KB Peak=41.12 KB
    HDFS_SCAN_NODE (id=0): Reservation=66.00 MB OtherMemory=1.26 MB Total=67.26 MB Peak=67.42 MB
      Exprs: Total=8.00 KB Peak=8.00 KB
    KrpcDataStreamSender (dst_id=5): Total=14.22 KB Peak=14.22 KB
  Fragment 4b497d5e4237002a:94c50a5200000050: Reservation=66.00 MB OtherMemory=51.47 MB Total=117.47 MB Peak=117.47 MB
    AGGREGATION_NODE (id=3): Total=46.45 MB Peak=48.23 MB
      NonGroupingAggregator 0: Total=8.00 KB Peak=8.00 KB
        Exprs: Total=4.00 KB Peak=4.00 KB
    HASH_JOIN_NODE (id=2): Reservation=0 OtherMemory=3.73 MB Total=3.73 MB Peak=8.86 MB
      Exprs: Total=41.12 KB Peak=41.12 KB
    HDFS_SCAN_NODE (id=0): Reservation=66.00 MB OtherMemory=1.26 MB Total=67.26 MB Peak=67.42 MB
      Exprs: Total=8.00 KB Peak=8.00 KB
    KrpcDataStreamSender (dst_id=5): Total=14.22 KB Peak=14.22 KB

The aggregation node is accumulating a lot of memory and it's a non-grouping agg that shouldn't accumulate much mem:

    AGGREGATION_NODE (id=3): Total=43.83 MB Peak=49.86 MB

Hit OOM on TPC-H Q19

Details

Description

Attachments

Activity

People

Dates