Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
None
-
ghx-label-1
Description
at org.apache.hadoop.cF0301 08:25:39.334376 21960 hdfs-lzo-text-scanner.cc:582] 744cd34df9463fba:610c3d1000000003] Check failed: stream_->eof() *** Check failure stack trace: *** @ 0x4e7bf5c google::LogMessage::Fail() @ 0x4e7d801 google::LogMessage::SendToLog() @ 0x4e7b936 google::LogMessage::Flush() @ 0x4e7eefd google::LogMessageFatal::~LogMessageFatal() @ 0x7fcf1248c2bd impala::HdfsLzoTextScanner::ReadAndDecompressData() @ 0x7fcf1248a0da impala::HdfsLzoTextScanner::ReadData() @ 0x7fcf1248a635 impala::HdfsLzoTextScanner::FillByteBuffer() @ 0x269f4ae impala::HdfsTextScanner::FillByteBufferWrapper() @ 0x269c047 impala::HdfsTextScanner::FinishScanRange() @ 0x269f272 impala::HdfsTextScanner::GetNextInternal() @ 0x7fcf12488b0d impala::HdfsLzoTextScanner::GetNextInternal() @ 0x2655808 impala::HdfsScanner::ProcessSplit() @ 0x27c1a4b impala::HdfsScanNode::ProcessSplit() @ 0x27c0c20 impala::HdfsScanNode::ScannerThread() @ 0x27bff7d _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv @ 0x27c2571 _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE @ 0x1fb39d5 boost::function0<>::operator()() @ 0x257402a impala::Thread::SuperviseThread() @ 0x257c2ae boost::_bi::list5<>::operator()<>() @ 0x257c1d2 boost::_bi::bind_t<>::operator()() @ 0x257c195 boost::detail::thread_data<>::run() @ 0x3da22f9 thread_proxy @ 0x7fcfdab10e24 start_thread @ 0x7fcfd767c34c __clone rypto.CryptoInputStream.read(CryptoInputStr
CORE: ./core.1583079954.31711.impalad BINARY: ./be/build/latest/service/impalad Core was generated by `/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f231918b1f7 in raise () from /lib64/libc.so.6 To enable execution of this file add add-auto-load-safe-path /data0/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/gcc-4.9.2/lib64/libstdc++.so.6.0.20-gdb.py line to your configuration file "/var/lib/jenkins/.gdbinit". To completely disable this security protection add set auto-load safe-path / line to your configuration file "/var/lib/jenkins/.gdbinit". For more information about this security protection see the "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: info "(gdb)Auto-loading safe path" #0 0x00007f231918b1f7 in raise () from /lib64/libc.so.6 #1 0x00007f231918c8e8 in abort () from /lib64/libc.so.6 #2 0x0000000004e85504 in google::DumpStackTraceAndExit() () #3 0x0000000004e7bf5d in google::LogMessage::Fail() () #4 0x0000000004e7d802 in google::LogMessage::SendToLog() () #5 0x0000000004e7b937 in google::LogMessage::Flush() () #6 0x0000000004e7eefe in google::LogMessageFatal::~LogMessageFatal() () #7 0x00007f2254b1d2be in impala::HdfsLzoTextScanner::ReadAndDecompressData (this=0x182062c0, pool=0xdb6cb68) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala-lzo/hdfs-lzo-text-scanner.cc:582 #8 0x00007f2254b1b0db in impala::HdfsLzoTextScanner::ReadData (this=0x182062c0, pool=0xdb6cb68) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala-lzo/hdfs-lzo-text-scanner.cc:307 #9 0x00007f2254b1b636 in impala::HdfsLzoTextScanner::FillByteBuffer (this=0x182062c0, pool=0xdb6cb68, eosr=0x7f224091f49e, num_bytes=65536) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala-lzo/hdfs-lzo-text-scanner.cc:342 #10 0x000000000269f4af in impala::HdfsTextScanner::FillByteBufferWrapper (this=0x182062c0, pool=0xdb6cb68, eosr=0x7f224091f49e, num_bytes=65536) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-text-scanner.cc:476 #11 0x000000000269c048 in impala::HdfsTextScanner::FinishScanRange (this=0x182062c0, row_batch=0xdb6cb40) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-text-scanner.cc:282 #12 0x000000000269f273 in impala::HdfsTextScanner::GetNextInternal (this=0x182062c0, row_batch=0xdb6cb40) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-text-scanner.cc:467 #13 0x00007f2254b19b0e in impala::HdfsLzoTextScanner::GetNextInternal (this=0x182062c0, row_batch=0xdb6cb40) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala-lzo/hdfs-lzo-text-scanner.cc:144 #14 0x0000000002655809 in impala::HdfsScanner::ProcessSplit (this=0x182062c0) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-scanner.cc:135 #15 0x00000000027c1a4c in impala::HdfsScanNode::ProcessSplit (this=0xa3d78d00, filter_ctxs=..., expr_results_pool=0x7f2240920410, scan_range=0x109d1f180, scanner_thread_reservation=0x7f2240920368) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-scan-node.cc:515 #16 0x00000000027c0c21 in impala::HdfsScanNode::ScannerThread (this=0xa3d78d00, first_thread=false, scanner_thread_reservation=8192) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-scan-node.cc:417 #17 0x00000000027bff7e in impala::HdfsScanNode::<lambda()>::operator()(void) const (__closure=0x7f2240920b98) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-scan-node.cc:338 #18 0x00000000027c2572 in boost::detail::function::void_function_obj_invoker0<impala::HdfsScanNode::ThreadTokenAvailableCb(impala::ThreadResourcePool*)::<lambda()>, void>::invoke(boost::detail::function::function_buffer &) (function_obj_ptr=...) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:159 #19 0x0000000001fb39d6 in boost::function0<void>::operator() (this=0x7f2240920b90) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:771 #20 0x000000000257402b in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*) (name=..., category=..., functor=..., parent_thread_info=0x7f2242720840, thread_started=0x7f224271f540) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/util/thread.cc:360 #21 0x000000000257c2af in boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) (this=0x1989013c0, f=@0x1989013b8: 0x2573cc4 <impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*)>, a=...) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:531 #22 0x000000000257c1d3 in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()() (this=0x1989013b8) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222 #23 0x000000000257c196 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > >::run() (this=0x198901200) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:116 #24 0x0000000003da22fa in thread_proxy () #25 0x00007f231c6e2e25 in start_thread () from /lib64/libpthread.so.0 #26 0x00007f231924e34d in clone () from /lib64/libc.so.6
This crashed all 3 impalads at around the same time:
Log file created at: 2020/03/01 08:26:07 Running on machine: impala-ec2-centos74-m5-4xlarge-ondemand-0a0f.vpc.cloudera.com Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg F0301 08:26:07.666946 27213 hdfs-lzo-text-scanner.cc:582] f44a5121256f5b7d:8e0bf92c00000001] Check failed: stream_->eof() Log file created at: 2020/03/01 08:25:53 Running on machine: impala-ec2-centos74-m5-4xlarge-ondemand-0a0f.vpc.cloudera.com Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg F0301 08:25:53.913563 25657 hdfs-lzo-text-scanner.cc:582] 1d4781605704837d:2a54525400000001] Check failed: stream_->eof() Log file created at: 2020/03/01 08:25:39 Running on machine: impala-ec2-centos74-m5-4xlarge-ondemand-0a0f.vpc.cloudera.com Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg F0301 08:25:39.334376 21960 hdfs-lzo-text-scanner.cc:582] 744cd34df9463fba:610c3d1000000003] Check failed: stream_->eof()
There was some noise in the log around the same time, unsure if it's relevant
hadoopZeroCopyRead: ZeroCopyCursor#read failed error: ReadOnlyBufferException: java.nio.ReadOnlyBufferException at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350) at org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660) at org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728) at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193) hadoopZeroCopyRead: ZeroCopyCursor#read failed error: ReadOnlyBufferException: java.nio.ReadOnlyBufferException at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350) at org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660) at org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728) at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193) hadoopZeroCopyRead: ZeroCopyCursor#read failed error: ReadOnlyBufferException: java.nio.ReadOnlyBufferException at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350) at org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660) at org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728) at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193) hadoopZeroCopyRead: ZeroCopyCursor#read failed error: ReadOnlyBufferException: java.nio.ReadOnlyBufferException at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350) at org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660) at org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728) at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193) hadoopZeroCopyRead: ZeroCopyCursor#read failed error: ReadOnlyBufferException: java.nio.ReadOnlyBufferException at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350) at org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660) at org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728) at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193) hadoopZeroCopyRead: ZeroCopyCursor#read failed error: ReadOnlyBufferException: java.nio.ReadOnlyBufferException at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350) at org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660) at org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728) at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193) hadoopZeroCopyRead: ZeroCopyCursor#read failed error: ReadOnlyBufferException: java.nio.ReadOnlyBufferException at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350) at org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660) at org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728) at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193) hadoopZeroCopyRead: ZeroCopyCursor#read failed error: ReadOnlyBufferException: java.nio.ReadOnlyBufferException at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350) at org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660) at org.apache.hadoop.cF0301 08:25:39.334376 21960 hdfs-lzo-text-scanner.cc:582] 744cd34df9463fba:610c3d1000000003] Check failed: stream_->eof() *** Check failure stack trace: ***
This was testing with a slightly old impala-lzo commit: https://github.com/cloudera/impala-lzo/commit/36d19a8a4559ec6703e5879684b0d341c75c0202
I'll attach logs and more info soon.