Tests failing with “Could not allocate memory while trying to increase reservation” on EC filesystem

In tests run against an Erasure Coding filesystem, multiple tests failed with memory allocation errors.

In total 10 tests failed:

• query_test.test_scanners.TestParquet.test_decimal_encodings
• query_test.test_scanners.TestTpchScanRangeLengths.test_tpch_scan_ranges
• query_test.test_exprs.TestExprs.test_exprs [enable_expr_rewrites: 0]
• query_test.test_exprs.TestExprs.test_exprs [enable_expr_rewrites: 1]
• query_test.test_hbase_queries.TestHBaseQueries.test_hbase_scan_node
• query_test.test_scanners.TestParquet.test_def_levels
• query_test.test_scanners.TestTextSplitDelimiters.test_text_split_across_buffers_delimiterquery_test.test_hbase_queries.TestHBaseQueries.test_hbase_filters
• query_test.test_hbase_queries.TestHBaseQueries.test_hbase_inline_views
• query_test.test_hbase_queries.TestHBaseQueries.test_hbase_top_n

The first failure looked like this on the client side:

F query_test/test_scanners.py::TestParquet:)::test_decimal_encodings[protocol: beeswax | exec_option:

Unknown macro: {'batch_size'}

| table_format: parquet/none]
query_test/test_scanners.py:717: in test_decimal_encodings
self.run_test_case('QueryTest/parquet-decimal-formats', vector, unique_database)
common/impala_test_suite.py:472: in run_test_case
result = self.__execute_query(target_impalad_client, query, user=user)
common/impala_test_suite.py:699: in __execute_query
return impalad_client.execute(query, user=user)
common/impala_connection.py:174: in execute
return self.__beeswax_client.execute(sql_stmt, user=user)
beeswax/impala_beeswax.py:183: in execute
handle = self.__execute_query(query_string.strip(), user=user)
beeswax/impala_beeswax.py:360: in __execute_query
self.wait_for_finished(handle)
beeswax/impala_beeswax.py:381: in wait_for_finished
raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
E ImpalaBeeswaxException: ImpalaBeeswaxException:
E Query aborted:ExecQueryFInstances rpc query_id=6e44c3c949a31be2:f973c7ff00000000 failed: Failed to get minimum memory reservation of 8.00 KB on daemon xxx.com:22001 for query 6e44c3c949a31be2:f973c7ff00000000 due to following error: Memory limit exceeded: Could not allocate memory while trying to increase reservation.
E Query(6e44c3c949a31be2:f973c7ff00000000) could not allocate 8.00 KB without exceeding limit.
E Error occurred on backend xxx.com:22001
E Memory left in process limit: 1.19 GB
E Query(6e44c3c949a31be2:f973c7ff00000000): Reservation=0 ReservationLimit=9.60 GB OtherMemory=0 Total=0 Peak=0
E Memory is likely oversubscribed. Reducing query concurrency or configuring admission control may help avoid this error.

On the server side log:

I0207 18:25:19.329311 5562 impala-server.cc:1063] 6e44c3c949a31be2:f973c7ff00000000] Registered query query_id=6e44c3c949a31be2:f973c7ff00000000 session_id=93497065f69e9d01:8a3bd06faff3da5
I0207 18:25:19.329434 5562 Frontend.java:1242] 6e44c3c949a31be2:f973c7ff00000000] Analyzing query: select score from decimal_stored_as_int32
I0207 18:25:19.329583 5562 FeSupport.java:285] 6e44c3c949a31be2:f973c7ff00000000] Requesting prioritized load of table(s): test_decimal_encodings_28d99c0e.decimal_stored_as_int32
I0207 18:25:30.776041 5562 Frontend.java:1282] 6e44c3c949a31be2:f973c7ff00000000] Analysis finished.
I0207 18:25:35.919486 10418 admission-controller.cc:608] 6e44c3c949a31be2:f973c7ff00000000] Schedule for id=6e44c3c949a31be2:f973c7ff00000000 in pool_name=default-pool per_host_mem_estimate=16.02 MB PoolConfig: max_requests=-1 max_queued=200 max_mem=-1.00 B
I0207 18:25:35.919528 10418 admission-controller.cc:613] 6e44c3c949a31be2:f973c7ff00000000] Stats: agg_num_running=2, agg_num_queued=0, agg_mem_reserved=24.13 MB, local_host(local_mem_admitted=1.99 GB, num_admitted_running=2, num_queued=0, backend_mem_reserved=8.06 MB)
I0207 18:25:35.919549 10418 admission-controller.cc:645] 6e44c3c949a31be2:f973c7ff00000000] Admitted query id=6e44c3c949a31be2:f973c7ff00000000
I0207 18:25:35.920532 10418 coordinator.cc:93] 6e44c3c949a31be2:f973c7ff00000000] Exec() query_id=6e44c3c949a31be2:f973c7ff00000000 stmt=select score from decimal_stored_as_int32
I0207 18:25:35.930855 10418 coordinator.cc:359] 6e44c3c949a31be2:f973c7ff00000000] starting execution on 2 backends for query_id=6e44c3c949a31be2:f973c7ff00000000
I0207 18:25:35.938108 21110 impala-internal-service.cc:50] 6e44c3c949a31be2:f973c7ff00000000] ExecQueryFInstances(): query_id=6e44c3c949a31be2:f973c7ff00000000 coord=xxx.com:22000 #instances=1
I0207 18:25:36.037228 10571 query-state.cc:624] 6e44c3c949a31be2:f973c7ff00000000] Executing instance. instance_id=6e44c3c949a31be2:f973c7ff00000000 fragment_idx=0 per_fragment_instance_idx=0 coord_state_idx=0 #in-flight=5
I0207 18:25:48.149771 12581 coordinator-backend-state.cc:209] ExecQueryFInstances rpc query_id=6e44c3c949a31be2:f973c7ff00000000 failed: Failed to get minimum memory reservation of 8.00 KB on daemon xxx.com:22001 for query 6e44c3c949a31be2:f973c7ff00000000 due to following error: Memory limit exceeded: Could not allocate memory while trying to increase reservation.
Query(6e44c3c949a31be2:f973c7ff00000000) could not allocate 8.00 KB without exceeding limit.
Query(6e44c3c949a31be2:f973c7ff00000000): Reservation=0 ReservationLimit=9.60 GB OtherMemory=0 Total=0 Peak=0
I0207 18:25:48.149895 10418 coordinator.cc:373] 6e44c3c949a31be2:f973c7ff00000000] started execution on 2 backends for query_id=6e44c3c949a31be2:f973c7ff00000000
I0207 18:25:48.152803 10418 coordinator.cc:527] 6e44c3c949a31be2:f973c7ff00000000] ExecState: query id=6e44c3c949a31be2:f973c7ff00000000 finstance=N/A on host=xxx.com (EXECUTING -> ERROR) status=ExecQueryFInstances rpc query_id=6e44c3c949a31be2:f973c7ff00000000 failed: Failed to get minimum memory reservation of 8.00 KB on daemon xxx.com:22001 for query 6e44c3c949a31be2:f973c7ff00000000 due to following error: Memory limit exceeded: Could not allocate memory while trying to increase reservation.
Query(6e44c3c949a31be2:f973c7ff00000000) could not allocate 8.00 KB without exceeding limit.
Query(6e44c3c949a31be2:f973c7ff00000000): Reservation=0 ReservationLimit=9.60 GB OtherMemory=0 Total=0 Peak=0
I0207 18:25:48.152827 10418 coordinator-backend-state.cc:453] 6e44c3c949a31be2:f973c7ff00000000] Sending CancelQueryFInstances rpc for query_id=6e44c3c949a31be2:f973c7ff00000000 backend=127.0.0.1:27000
I0207 18:25:48.155086 12737 control-service.cc:168] CancelQueryFInstances(): query_id=6e44c3c949a31be2:f973c7ff00000000
I0207 18:25:48.155109 12737 query-exec-mgr.cc:97] QueryState: query_id=6e44c3c949a31be2:f973c7ff00000000 refcnt=4
I0207 18:25:48.155117 12737 query-state.cc:649] Cancel: query_id=6e44c3c949a31be2:f973c7ff00000000
I0207 18:25:48.155129 12737 krpc-data-stream-mgr.cc:325] cancelling all streams for fragment_instance_id=6e44c3c949a31be2:f973c7ff00000000
I0207 18:25:48.155297 10418 coordinator.cc:687] 6e44c3c949a31be2:f973c7ff00000000] CancelBackends() query_id=6e44c3c949a31be2:f973c7ff00000000, tried to cancel 1 backends
I0207 18:25:48.155306 10418 coordinator.cc:859] 6e44c3c949a31be2:f973c7ff00000000] Release admission control resources for query_id=6e44c3c949a31be2:f973c7ff00000000
I0207 18:25:48.170018 10571 krpc-data-stream-mgr.cc:294] 6e44c3c949a31be2:f973c7ff00000000] DeregisterRecvr(): fragment_instance_id=6e44c3c949a31be2:f973c7ff00000000, node=1
I0207 18:25:48.197767 5562 impala-beeswax-server.cc:239] close(): query_id=6e44c3c949a31be2:f973c7ff00000000
I0207 18:25:48.197775 5562 impala-server.cc:1142] UnregisterQuery(): query_id=6e44c3c949a31be2:f973c7ff00000000
I0207 18:25:48.197779 5562 impala-server.cc:1249] Cancel(): query_id=6e44c3c949a31be2:f973c7ff00000000
I0207 18:25:48.225905 10529 query-state.cc:272] 6e44c3c949a31be2:f973c7ff00000000] UpdateBackendExecState(): last report for 6e44c3c949a31be2:f973c7ff00000000
I0207 18:25:48.225889 10571 query-state.cc:632] 6e44c3c949a31be2:f973c7ff00000000] Instance completed. instance_id=6e44c3c949a31be2:f973c7ff00000000 #in-flight=4 status=CANCELLED: Cancelled
I0207 18:25:48.372977 12737 control-service.cc:125] ReportExecStatus(): Received report for unknown query ID (probably closed or cancelled): 6e44c3c949a31be2:f973c7ff00000000 remote host=127.0.0.1:50422
I0207 18:25:48.373118 10529 query-state.cc:431] 6e44c3c949a31be2:f973c7ff00000000] Cancelling fragment instances as directed by the coordinator. Returned status: ReportExecStatus(): Received report for unknown query ID (probably closed or cancelled): 6e44c3c949a31be2:f973c7ff00000000 remote host=127.0.0.1:50422
I0207 18:25:48.373138 10529 query-state.cc:649] 6e44c3c949a31be2:f973c7ff00000000] Cancel: query_id=6e44c3c949a31be2:f973c7ff00000000
I0207 18:25:48.429422 5562 query-exec-mgr.cc:184] ReleaseQueryState(): deleted query_id=6e44c3c949a31be2:f973c7ff00000000