Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5773

Memory limit exceeded on test_spilling.py

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • Impala 2.7.0, Impala 2.8.0, Impala 2.9.0, Impala 2.10.0
    • Impala 2.10.0
    • Backend

    Description

      03:55:47 FAIL query_test/test_spilling.py::TestSpilling::()::test_spilling[exec_option: {'default_spillable_buffer_size': '256k'} | table_format: parquet/none]
03:55:47 =================================== FAILURES ===================================
03:55:47  TestSpilling.test_spilling[exec_option: {'default_spillable_buffer_size': '256k'} | table_format: parquet/none] 
03:55:47 [gw1] linux2 -- Python 2.6.6 /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/bin/../infra/python/env/bin/python
03:55:47 query_test/test_spilling.py:39: in test_spilling
03:55:47     self.run_test_case('QueryTest/spilling', vector)
03:55:47 common/impala_test_suite.py:390: in run_test_case
03:55:47     result = self.__execute_query(target_impalad_client, query, user=user)
03:55:47 common/impala_test_suite.py:598: in __execute_query
03:55:47     return impalad_client.execute(query, user=user)
03:55:47 common/impala_connection.py:160: in execute
03:55:47     return self.__beeswax_client.execute(sql_stmt, user=user)
03:55:47 beeswax/impala_beeswax.py:173: in execute
03:55:47     handle = self.__execute_query(query_string.strip(), user=user)
03:55:47 beeswax/impala_beeswax.py:339: in __execute_query
03:55:47     self.wait_for_completion(handle)
03:55:47 beeswax/impala_beeswax.py:359: in wait_for_completion
03:55:47     raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
03:55:47 E   ImpalaBeeswaxException: ImpalaBeeswaxException:
03:55:47 E    Query aborted:Memory limit exceeded: Error occurred on backend impala-boost-static-burst-slave-1c37.vpc.cloudera.com:22000 by fragment be415e2081bde55d:ce5cb0b400000001
03:55:47 E   Memory left in process limit: 49.35 GB
03:55:47 E   Memory left in query limit: -36546.00 B
03:55:47 E   Query(be415e2081bde55d:ce5cb0b400000000): memory limit exceeded. Limit=800.00 MB Reservation=640.00 MB ReservationLimit=640.00 MB OtherMemory=160.03 MB Total=800.03 MB Peak=800.03 MB
03:55:47 E     Fragment be415e2081bde55d:ce5cb0b400000000: Reservation=0 OtherMemory=12.24 KB Total=12.24 KB Peak=63.50 KB
03:55:47 E       AGGREGATION_NODE (id=6): Total=4.00 KB Peak=4.00 KB
03:55:47 E         Exprs: Total=4.00 KB Peak=4.00 KB
03:55:47 E       EXCHANGE_NODE (id=5): Total=0 Peak=0
03:55:47 E       DataStreamRecvr: Total=0 Peak=0
03:55:47 E       PLAN_ROOT_SINK: Total=0 Peak=0
03:55:47 E       CodeGen: Total=247.00 B Peak=51.50 KB
03:55:47 E     Fragment be415e2081bde55d:ce5cb0b400000002: Reservation=24.50 MB OtherMemory=157.98 MB Total=182.48 MB Peak=182.48 MB
03:55:47 E       AGGREGATION_NODE (id=2): Total=4.00 KB Peak=4.00 KB
03:55:47 E         Exprs: Total=4.00 KB Peak=4.00 KB
03:55:47 E       AGGREGATION_NODE (id=4): Reservation=24.50 MB OtherMemory=29.12 KB Total=24.53 MB Peak=25.80 MB
03:55:47 E         Exprs: Total=21.12 KB Peak=21.12 KB
03:55:47 E       EXCHANGE_NODE (id=3): Total=0 Peak=0
03:55:47 E       DataStreamRecvr: Total=157.20 MB Peak=157.20 MB
03:55:47 E       DataStreamSender (dst_id=5): Total=16.00 KB Peak=16.00 KB
03:55:47 E       CodeGen: Total=5.09 KB Peak=373.50 KB
03:55:47 E     Fragment be415e2081bde55d:ce5cb0b400000001: Reservation=615.50 MB OtherMemory=2.04 MB Total=617.54 MB Peak=651.88 MB
03:55:47 E       AGGREGATION_NODE (id=1): Reservation=615.50 MB OtherMemory=2.02 MB Total=617.52 MB Peak=618.70 MB
03:55:47 E         Exprs: Total=2.02 MB Peak=2.02 MB
03:55:47 E       HDFS_SCAN_NODE (id=0): Total=0 Peak=33.32 MB
03:55:47 E       DataStreamSender (dst_id=3): Total=7.52 KB Peak=7.52 KB
03:55:47 E       CodeGen: Total=6.84 KB Peak=522.50 KB
03:55:47 E   
03:55:47 E   Memory limit exceeded: Error occurred on backend impala-boost-static-burst-slave-1c37.vpc.cloudera.com:22000 by fragment be415e2081bde55d:ce5cb0b400000001
03:55:47 E   Memory left in process limit: 49.35 GB
03:55:47 E   Memory left in query limit: -36546.00 B
03:55:47 E   Query(be415e2081bde55d:ce5cb0b400000000): memory limit exceeded. Limit=800.00 MB Reservation=640.00 MB ReservationLimit=640.00 MB OtherMemory=160.03 MB Total=800.03 MB Peak=800.03 MB
03:55:47 E     Fragment be415e2081bde55d:ce5cb0b400000000: Reservation=0 OtherMemory=12.24 KB Total=12.24 KB Peak=63.50 KB
03:55:47 E       AGGREGATION_NODE (id=6): Total=4.00 KB Peak=4.00 KB
03:55:47 E         Exprs: Total=4.00 KB Peak=4.00 KB
03:55:47 E       EXCHANGE_NODE (id=5): Total=0 Peak=0
03:55:47 E       DataStreamRecvr: Total=0 Peak=0
03:55:47 E       PLAN_ROOT_SINK: Total=0 Peak=0
03:55:47 E       CodeGen: Total=247.00 B Peak=51.50 KB
03:55:47 E     Fragment be415e2081bde55d:ce5cb0b400000002: Reservation=24.50 MB OtherMemory=157.98 MB Total=182.48 MB Peak=182.48 MB
03:55:47 E       AGGREGATION_NODE (id=2): Total=4.00 KB Peak=4.00 KB
03:55:47 E         Exprs: Total=4.00 KB Peak=4.00 KB
03:55:47 E       AGGREGATION_NODE (id=4): Reservation=24.50 MB OtherMemory=29.12 KB Total=24.53 MB Peak=25.80 MB
03:55:47 E         Exprs: Total=21.12 KB Peak=21.12 KB
03:55:47 E       EXCHANGE_NODE (id=3): Total=0 Peak=0
03:55:47 E       DataStreamRecvr: Total=157.20 MB Peak=157.20 MB
03:55:47 E       DataStreamSender (dst_id=5): Total=16.00 KB Peak=16.00 KB
03:55:47 E       CodeGen: Total=5.09 KB Peak=373.50 KB
03:55:47 E     Fragment be415e2081bde55d:ce5cb0b400000001: Reservation=615.50 MB OtherMemory=2.04 MB Total=617.54 MB Peak=651.88 MB
03:55:47 E       AGGREGATION_NODE (id=1): Reservation=615.50 MB OtherMemory=2.02 MB Total=617.52 MB Peak=618.70 MB
03:55:47 E         Exprs: Total=2.02 MB Peak=2.02 MB
03:55:47 E       HDFS_SCAN_NODE (id=0): Total=0 Peak=33.32 MB
03:55:47 E       DataStreamSender (dst_id=3): Total=7.52 KB Peak=7.52 KB
03:55:47 E       CodeGen: Total=6.84 KB Peak=522.50 KB (1 of 2 similar)
03:55:47 ---------------------------- Captured stderr setup -----------------------------
03:55:47 -- connecting to: localhost:21000
03:55:47 ----------------------------- Captured stderr call -----------------------------
03:55:47 -- executing against localhost:21000
03:55:47 use tpch_parquet;
03:55:47 
03:55:47 SET default_spillable_buffer_size=256k;
03:55:47 -- executing against localhost:21000
03:55:47 set buffer_pool_limit=10m;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 
03:55:47 select l_orderkey, count(*)
03:55:47 from lineitem
03:55:47 group by 1
03:55:47 order by 1 limit 10;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 SET BUFFER_POOL_LIMIT=0;;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 set buffer_pool_limit=10m;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 
03:55:47 set num_nodes=1;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 
03:55:47 select l_returnflag, l_orderkey, avg(l_tax), min(l_shipmode)
03:55:47 from lineitem
03:55:47 group by 1,2
03:55:47 order by 1,2 limit 3;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 SET BUFFER_POOL_LIMIT=0;;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 SET NUM_NODES=0;;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 set buffer_pool_limit=10m;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 
03:55:47 select l_orderkey, count(*)
03:55:47 from lineitem
03:55:47 group by 1
03:55:47 order by 1 limit 10;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 SET BUFFER_POOL_LIMIT=0;;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 set buffer_pool_limit=10m;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 
03:55:47 set num_nodes=1;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 
03:55:47 select l_comment, count(*)
03:55:47 from lineitem
03:55:47 group by 1
03:55:47 order by count(*) desc limit 5;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 SET BUFFER_POOL_LIMIT=0;;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 SET NUM_NODES=0;;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 set buffer_pool_limit=10m;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 
03:55:47 set num_nodes=1;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 
03:55:47 select l_returnflag, l_orderkey, round(avg(l_tax),2), min(l_shipmode)
03:55:47 from lineitem
03:55:47 group by 1,2
03:55:47 order by 1,2 limit 3;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 SET BUFFER_POOL_LIMIT=0;;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 SET NUM_NODES=0;;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 set buffer_pool_limit=10m;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 
03:55:47 select l_orderkey, avg(l_orderkey)
03:55:47 from lineitem
03:55:47 group by 1
03:55:47 order by 1 limit 5;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 SET BUFFER_POOL_LIMIT=0;;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 set buffer_pool_limit=15m;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 
03:55:47 select count(l1.l_tax)
03:55:47 from
03:55:47 lineitem l1,
03:55:47 lineitem l2,
03:55:47 lineitem l3
03:55:47 where
03:55:47 l1.l_tax < 0.01 and
03:55:47 l2.l_tax < 0.04 and
03:55:47 l1.l_orderkey = l2.l_orderkey and
03:55:47 l1.l_orderkey = l3.l_orderkey and
03:55:47 l1.l_comment = l3.l_comment and
03:55:47 l1.l_shipdate = l3.l_shipdate;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 SET BUFFER_POOL_LIMIT=0;;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 set buffer_pool_limit=2m;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 
03:55:47 select max(t1.total_count), max(t1.l_shipinstruct), max(t1.l_comment) from
03:55:47 (select l_shipinstruct, l_comment, count(*) over () total_count from lineitem) t1;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 SET BUFFER_POOL_LIMIT=0;;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 set buffer_pool_limit=20m;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 
03:55:47 select a.int_col, count(*)
03:55:47 from functional.alltypessmall a, functional.alltypessmall b, functional.alltypessmall c
03:55:47 where a.id = b.id and b.id = c.id group by a.int_col;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 SET BUFFER_POOL_LIMIT=0;;
03:55:47 
03:55:47 MainThread: Query Name: 
03:55:47 TPCH-Q22
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 set buffer_pool_limit=20m;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 
03:55:47 select
03:55:47   s_name,
03:55:47   count(*) as numwait
03:55:47 from
03:55:47   supplier,
03:55:47   lineitem l1 join [BROADCAST]
03:55:47   orders,
03:55:47   nation
03:55:47 where
03:55:47   s_suppkey = l1.l_suppkey
03:55:47   and o_orderkey = l1.l_orderkey
03:55:47   and o_orderstatus = 'F'
03:55:47   and l1.l_receiptdate > l1.l_commitdate
03:55:47   and exists (
03:55:47     select
03:55:47       *
03:55:47     from
03:55:47       lineitem l2
03:55:47     where
03:55:47       l2.l_orderkey = l1.l_orderkey
03:55:47       and l2.l_suppkey <> l1.l_suppkey
03:55:47   )
03:55:47   and not exists (
03:55:47     select
03:55:47       *
03:55:47     from
03:55:47       lineitem l3
03:55:47     where
03:55:47       l3.l_orderkey = l1.l_orderkey
03:55:47       and l3.l_suppkey <> l1.l_suppkey
03:55:47       and l3.l_receiptdate > l3.l_commitdate
03:55:47   )
03:55:47   and s_nationkey = n_nationkey
03:55:47   and n_name = 'SAUDI ARABIA'
03:55:47 group by
03:55:47   s_name
03:55:47 order by
03:55:47   numwait desc,
03:55:47   s_name
03:55:47 limit 100;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 SET BUFFER_POOL_LIMIT=0;;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 set buffer_pool_limit=50m;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 
03:55:47 select l_orderkey, count(*), group_concat(distinct l_linestatus, '|')
03:55:47 from lineitem
03:55:47 group by 1
03:55:47 order by 1 limit 10;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 SET BUFFER_POOL_LIMIT=0;;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 set mem_limit=800m;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 
03:55:47 set num_scanner_threads=1;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 
03:55:47 select count(distinct concat(cast(l_comment as char(120)), cast(l_comment as char(120)),
03:55:47                              cast(l_comment as char(120)), cast(l_comment as char(120)),
03:55:47                              cast(l_comment as char(120)), cast(l_comment as char(120))))
03:55:47 from lineitem;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 SET MEM_LIMIT=0;;
03:55:47 
03:55:47 -- executing against localhost:21000
03:55:47 SET NUM_SCANNER_THREADS=0;;
03:55:47 
03:55:47 ======= 1 failed, 1323 passed, 174 skipped, 4 xfailed in 3454.29 seconds =======
03:55:47 ============================= test session starts ==============================

      Tim Armstrong, can you please take a first look ?

      Attachments

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            henryr Henry Robinson
            kwho Michael Ho
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment