Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
Impala 2.0, Impala 2.1
-
None
Description
After fixing some of the low mem / spilling issues, the simple low mem stress test found another problem: In the infrequent case where at Partition::hash_tbl::Init() couldn't consume 8KB to allocate the buckets array it would return false. But then, PAGG::Partition::Close() would try to clean up the hash table iterating the buckets. even though it didn't have any buckets. That would cause a dcheck() in HT::Begin().
The HashTable::Init() code would return set num_buckets_ == 0 and return false if it couldn't consume 8KB.
bool HashTable::Init() {
int64_t buckets_byte_size = num_buckets_ * sizeof(Bucket);
if (block_mgr_client_ != NULL &&
!state_->block_mgr()->ConsumeMemory(block_mgr_client_, buckets_byte_size)) {
num_buckets_ = 0; <-- couldn't consume 8KB, set num_buckets_ == 0.
return false;
}
buckets_ = reinterpret_cast<Bucket*>(malloc(buckets_byte_size));
memset(buckets_, 0, buckets_byte_size);
return GrowNodeArray();
}
The PAGG::Partition::Close() would try to iterate the hash_tbl even though it didn't have any buckets.
void PartitionedAggregationNode::Partition::Close(bool finalize_rows) {
if (is_closed) return;
is_closed = true;
if (aggregated_row_stream.get() != NULL) {
if (finalize_rows && hash_tbl.get() != NULL) {
// We need to walk all the rows and Finalize them here so the UDA gets a chance
// to cleanup. If the hash table is gone (meaning this was spilled), the rows
// should have been finalized/serialized in Spill().
parent->CleanupHashTbl(agg_fn_ctxs, hash_tbl->Begin(parent->ht_ctx_.get())); <--- the hash_tbl->Begin() call will DCHECK if num_buckets_ == 0
}
aggregated_row_stream->Close();
}