Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Cannot Reproduce
-
Impala 2.3.0
Description
Vtune profile data shows that BlockingQueue is a potential bottleneck in the scan code.
BlockingQueue should be replaced with a queue that allows readers and writers to operate concurrently.
Data Of Interest (CPU Metrics) 2 of 14: 30.6% (1.309s of 4.279s) libpthread.so.0!pthread_cond_signal - [Unknown] impalad!impala::ExecNode::RowBatchQueue::GetBatch+0x158 - [Unknown]:[Unknown] impalad!impala::HdfsScanNode::GetNextInternal+0xf0 - [Unknown]:[Unknown] impalad!impala::HdfsScanNode::GetNext+0x2fc - [Unknown]:[Unknown] impalad!impala::PartitionedAggregationNode::Open+0x344 - [Unknown]:[Unknown] impalad!impala::PlanFragmentExecutor::OpenInternal+0x92 - [Unknown]:[Unknown] impalad!impala::PlanFragmentExecutor::Open+0x2d8 - [Unknown]:[Unknown] impalad!impala::FragmentMgr::FragmentExecState::Exec+0x1f - [Unknown]:[Unknown] impalad!impala::FragmentMgr::FragmentExecThread+0x4b - [Unknown]:[Unknown] impalad!impala::Thread::SuperviseThread+0x1b9 - [Unknown]:[Unknown] impalad!boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void (void)>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void (void)>>, boost::_bi::value<impala::Promise<long>*>>>>::run+0x7f - [Unknown]:[Unknown] impalad!func@0xd18ce0+0x62 - [Unknown]:[Unknown] libpthread.so.0!start_thread+0xd0 - [Unknown]:[Unknown] libc.so.6!clone+0x6c - [Unknown]:[Unknown]
Currently the Queue size is based on, when working on the new version of the queue this should be revisited
10 * (DiskInfo::num_disks() + DiskIoMgr::REMOTE_NUM_DISKS)
RowBatch* ExecNode::RowBatchQueue::GetBatch() { RowBatch* result = NULL; if (BlockingGet(&result)) return result; return NULL; }
Attachments
Issue Links
- is related to
-
IMPALA-4026 Investigate regression introduced by "IMPALA-3629: Codegen TransferScratchTuples"
- Resolved
-
IMPALA-2712 Investigate increasing batch size
- Resolved