Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
Impala 2.3.0
-
None
-
impalad version 2.3.0-cdh5-INTERNAL DEBUG (build c21a2032f6b9b6a0861993372da9d749a3e905d1)
Built on Thu, 08 Oct 2015 01:58:44 PST
Description
After ~4 hrs and ~10k queries a stress run crashed saying
impala-stress-5.vpc.cloudera.com crashed: F1008 18:38:58.361202 10472 buffered-block-mgr.cc:891] Check failed: Validate() Buffered block mgr Num writes outstanding: 0 Num free io buffers: 5 Num unpinned blocks: 35 Num available buffers: -6 Total pinned buffers: 454 Unfullfilled reserved buffers: 334 Remaining memory: 2422471680 (#blocks=288) Block write threshold: 4 [...skipped...] #6 0x000000000205f4ed in google::LogMessageFatal::~LogMessageFatal (this=0x7f8fdcb03280, __in_chrg=<value optimized out>) at src/logging.cc:1836 #7 0x0000000001231ba7 in impala::BufferedBlockMgr::DeleteBlock (this=0x7cb7c00, block=0x568c06c0) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/runtime/buffered-block-mgr.cc:891 #8 0x000000000122993e in impala::BufferedBlockMgr::Block::Delete (this=0x568c06c0) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/runtime/buffered-block-mgr.cc:133 #9 0x0000000001574135 in impala::BufferedTupleStream::Close (this=0xc773b560) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/runtime/buffered-tuple-stream.cc:159 #10 0x000000000168d012 in impala::PartitionedHashJoinNode::BuildHashTables (this=0x6cc0fb00, state=0x1b3346d00) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/exec/partitioned-hash-join-node.cc:1237 #11 0x000000000168757d in impala::PartitionedHashJoinNode::ProcessBuildInput (this=0x6cc0fb00, state=0x1b3346d00, level=1) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/exec/partitioned-hash-join-node.cc:666 #12 0x000000000168891d in impala::PartitionedHashJoinNode::PrepareNextPartition (this=0x6cc0fb00, state=0x1b3346d00) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/exec/partitioned-hash-join-node.cc:781 #13 0x000000000168a253 in impala::PartitionedHashJoinNode::GetNext (this=0x6cc0fb00, state=0x1b3346d00, out_batch=0x1d8dd5860, eos=0x6cc11be1) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/exec/partitioned-hash-join-node.cc:941 #14 0x000000000168787f in impala::PartitionedHashJoinNode::NextProbeRowBatch (this=0x6cc11a80, state=0x1b3346d00, out_batch=0xec5932c0) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/exec/partitioned-hash-join-node.cc:692 #15 0x0000000001689f47 in impala::PartitionedHashJoinNode::GetNext (this=0x6cc11a80, state=0x1b3346d00, out_batch=0xec5932c0, eos=0x3d0165361) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/exec/partitioned-hash-join-node.cc:916 #16 0x000000000168787f in impala::PartitionedHashJoinNode::NextProbeRowBatch (this=0x3d0165200, state=0x1b3346d00, out_batch=0x6c4574a0) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/exec/partitioned-hash-join-node.cc:692 #17 0x0000000001689f47 in impala::PartitionedHashJoinNode::GetNext (this=0x3d0165200, state=0x1b3346d00, out_batch=0x6c4574a0, eos=0x3d0167761) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/exec/partitioned-hash-join-node.cc:916 #18 0x000000000168787f in impala::PartitionedHashJoinNode::NextProbeRowBatch (this=0x3d0167600, state=0x1b3346d00, out_batch=0x7f8fdcb04b20) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/exec/partitioned-hash-join-node.cc:692 #19 0x0000000001689f47 in impala::PartitionedHashJoinNode::GetNext (this=0x3d0167600, state=0x1b3346d00, out_batch=0x7f8fdcb04b20, eos=0x7f8fdcb04d3f) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/exec/partitioned-hash-join-node.cc:916 #20 0x000000000166eba7 in impala::PartitionedAggregationNode::Open (this=0x28844c600, state=0x1b3346d00) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/exec/partitioned-aggregation-node.cc:240 #21 0x000000000155fc4b in impala::PlanFragmentExecutor::OpenInternal (this=0x1e2d7dd28) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/runtime/plan-fragment-executor.cc:334 #22 0x000000000155faec in impala::PlanFragmentExecutor::Open (this=0x1e2d7dd28) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/runtime/plan-fragment-executor.cc:320 #23 0x0000000001333060 in impala::FragmentMgr::FragmentExecState::Exec (this=0x1e2d7db00) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/service/fragment-exec-state.cc:50 #24 0x000000000132b67e in impala::FragmentMgr::FragmentExecThread (this=0x5b92ea0, exec_state=0x1e2d7db00) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/service/fragment-mgr.cc:70
I'll post the location of the logs/core dump when it is available. The cluster will be up until tomorrow if anyone wants to look now. The crashed node was impala-stress-5.vpc.cloudera.com.