Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-9449

Crash in LZO scan with fuzz test

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      	at org.apache.hadoop.cF0301 08:25:39.334376 21960 hdfs-lzo-text-scanner.cc:582] 744cd34df9463fba:610c3d1000000003] Check failed: stream_->eof() 
      *** Check failure stack trace: ***
          @          0x4e7bf5c  google::LogMessage::Fail()
          @          0x4e7d801  google::LogMessage::SendToLog()
          @          0x4e7b936  google::LogMessage::Flush()
          @          0x4e7eefd  google::LogMessageFatal::~LogMessageFatal()
          @     0x7fcf1248c2bd  impala::HdfsLzoTextScanner::ReadAndDecompressData()
          @     0x7fcf1248a0da  impala::HdfsLzoTextScanner::ReadData()
          @     0x7fcf1248a635  impala::HdfsLzoTextScanner::FillByteBuffer()
          @          0x269f4ae  impala::HdfsTextScanner::FillByteBufferWrapper()
          @          0x269c047  impala::HdfsTextScanner::FinishScanRange()
          @          0x269f272  impala::HdfsTextScanner::GetNextInternal()
          @     0x7fcf12488b0d  impala::HdfsLzoTextScanner::GetNextInternal()
          @          0x2655808  impala::HdfsScanner::ProcessSplit()
          @          0x27c1a4b  impala::HdfsScanNode::ProcessSplit()
          @          0x27c0c20  impala::HdfsScanNode::ScannerThread()
          @          0x27bff7d  _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
          @          0x27c2571  _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
          @          0x1fb39d5  boost::function0<>::operator()()
          @          0x257402a  impala::Thread::SuperviseThread()
          @          0x257c2ae  boost::_bi::list5<>::operator()<>()
          @          0x257c1d2  boost::_bi::bind_t<>::operator()()
          @          0x257c195  boost::detail::thread_data<>::run()
          @          0x3da22f9  thread_proxy
          @     0x7fcfdab10e24  start_thread
          @     0x7fcfd767c34c  __clone
      rypto.CryptoInputStream.read(CryptoInputStr
      
      CORE: ./core.1583079954.31711.impalad
      BINARY: ./be/build/latest/service/impalad
      Core was generated by `/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be'.
      Program terminated with signal SIGABRT, Aborted.
      #0  0x00007f231918b1f7 in raise () from /lib64/libc.so.6
      To enable execution of this file add
      	add-auto-load-safe-path /data0/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/gcc-4.9.2/lib64/libstdc++.so.6.0.20-gdb.py
      line to your configuration file "/var/lib/jenkins/.gdbinit".
      To completely disable this security protection add
      	set auto-load safe-path /
      line to your configuration file "/var/lib/jenkins/.gdbinit".
      For more information about this security protection see the
      "Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
      	info "(gdb)Auto-loading safe path"
      #0  0x00007f231918b1f7 in raise () from /lib64/libc.so.6
      #1  0x00007f231918c8e8 in abort () from /lib64/libc.so.6
      #2  0x0000000004e85504 in google::DumpStackTraceAndExit() ()
      #3  0x0000000004e7bf5d in google::LogMessage::Fail() ()
      #4  0x0000000004e7d802 in google::LogMessage::SendToLog() ()
      #5  0x0000000004e7b937 in google::LogMessage::Flush() ()
      #6  0x0000000004e7eefe in google::LogMessageFatal::~LogMessageFatal() ()
      #7  0x00007f2254b1d2be in impala::HdfsLzoTextScanner::ReadAndDecompressData (this=0x182062c0, pool=0xdb6cb68) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala-lzo/hdfs-lzo-text-scanner.cc:582
      #8  0x00007f2254b1b0db in impala::HdfsLzoTextScanner::ReadData (this=0x182062c0, pool=0xdb6cb68) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala-lzo/hdfs-lzo-text-scanner.cc:307
      #9  0x00007f2254b1b636 in impala::HdfsLzoTextScanner::FillByteBuffer (this=0x182062c0, pool=0xdb6cb68, eosr=0x7f224091f49e, num_bytes=65536) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala-lzo/hdfs-lzo-text-scanner.cc:342
      #10 0x000000000269f4af in impala::HdfsTextScanner::FillByteBufferWrapper (this=0x182062c0, pool=0xdb6cb68, eosr=0x7f224091f49e, num_bytes=65536) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-text-scanner.cc:476
      #11 0x000000000269c048 in impala::HdfsTextScanner::FinishScanRange (this=0x182062c0, row_batch=0xdb6cb40) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-text-scanner.cc:282
      #12 0x000000000269f273 in impala::HdfsTextScanner::GetNextInternal (this=0x182062c0, row_batch=0xdb6cb40) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-text-scanner.cc:467
      #13 0x00007f2254b19b0e in impala::HdfsLzoTextScanner::GetNextInternal (this=0x182062c0, row_batch=0xdb6cb40) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala-lzo/hdfs-lzo-text-scanner.cc:144
      #14 0x0000000002655809 in impala::HdfsScanner::ProcessSplit (this=0x182062c0) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-scanner.cc:135
      #15 0x00000000027c1a4c in impala::HdfsScanNode::ProcessSplit (this=0xa3d78d00, filter_ctxs=..., expr_results_pool=0x7f2240920410, scan_range=0x109d1f180, scanner_thread_reservation=0x7f2240920368) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-scan-node.cc:515
      #16 0x00000000027c0c21 in impala::HdfsScanNode::ScannerThread (this=0xa3d78d00, first_thread=false, scanner_thread_reservation=8192) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-scan-node.cc:417
      #17 0x00000000027bff7e in impala::HdfsScanNode::<lambda()>::operator()(void) const (__closure=0x7f2240920b98) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-scan-node.cc:338
      #18 0x00000000027c2572 in boost::detail::function::void_function_obj_invoker0<impala::HdfsScanNode::ThreadTokenAvailableCb(impala::ThreadResourcePool*)::<lambda()>, void>::invoke(boost::detail::function::function_buffer &) (function_obj_ptr=...) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:159
      #19 0x0000000001fb39d6 in boost::function0<void>::operator() (this=0x7f2240920b90) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:771
      #20 0x000000000257402b in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*) (name=..., category=..., functor=..., parent_thread_info=0x7f2242720840, thread_started=0x7f224271f540) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/util/thread.cc:360
      #21 0x000000000257c2af in boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) (this=0x1989013c0, f=@0x1989013b8: 0x2573cc4 <impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*)>, a=...) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:531
      #22 0x000000000257c1d3 in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()() (this=0x1989013b8) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222
      #23 0x000000000257c196 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > >::run() (this=0x198901200) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:116
      #24 0x0000000003da22fa in thread_proxy ()
      #25 0x00007f231c6e2e25 in start_thread () from /lib64/libpthread.so.0
      #26 0x00007f231924e34d in clone () from /lib64/libc.so.6
      

      This crashed all 3 impalads at around the same time:

      Log file created at: 2020/03/01 08:26:07
      Running on machine: impala-ec2-centos74-m5-4xlarge-ondemand-0a0f.vpc.cloudera.com
      Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
      F0301 08:26:07.666946 27213 hdfs-lzo-text-scanner.cc:582] f44a5121256f5b7d:8e0bf92c00000001] Check failed: stream_->eof() 
      
      Log file created at: 2020/03/01 08:25:53
      Running on machine: impala-ec2-centos74-m5-4xlarge-ondemand-0a0f.vpc.cloudera.com
      Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
      F0301 08:25:53.913563 25657 hdfs-lzo-text-scanner.cc:582] 1d4781605704837d:2a54525400000001] Check failed: stream_->eof() 
      
      Log file created at: 2020/03/01 08:25:39
      Running on machine: impala-ec2-centos74-m5-4xlarge-ondemand-0a0f.vpc.cloudera.com
      Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
      F0301 08:25:39.334376 21960 hdfs-lzo-text-scanner.cc:582] 744cd34df9463fba:610c3d1000000003] Check failed: stream_->eof() 
      

      There was some noise in the log around the same time, unsure if it's relevant

      hadoopZeroCopyRead: ZeroCopyCursor#read failed error:
      ReadOnlyBufferException: java.nio.ReadOnlyBufferException
      	at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350)
      	at org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660)
      	at org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728)
      	at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193)
      hadoopZeroCopyRead: ZeroCopyCursor#read failed error:
      ReadOnlyBufferException: java.nio.ReadOnlyBufferException
      	at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350)
      	at org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660)
      	at org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728)
      	at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193)
      hadoopZeroCopyRead: ZeroCopyCursor#read failed error:
      ReadOnlyBufferException: java.nio.ReadOnlyBufferException
      	at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350)
      	at org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660)
      	at org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728)
      	at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193)
      hadoopZeroCopyRead: ZeroCopyCursor#read failed error:
      ReadOnlyBufferException: java.nio.ReadOnlyBufferException
      	at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350)
      	at org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660)
      	at org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728)
      	at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193)
      hadoopZeroCopyRead: ZeroCopyCursor#read failed error:
      ReadOnlyBufferException: java.nio.ReadOnlyBufferException
      	at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350)
      	at org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660)
      	at org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728)
      	at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193)
      hadoopZeroCopyRead: ZeroCopyCursor#read failed error:
      ReadOnlyBufferException: java.nio.ReadOnlyBufferException
      	at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350)
      	at org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660)
      	at org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728)
      	at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193)
      hadoopZeroCopyRead: ZeroCopyCursor#read failed error:
      ReadOnlyBufferException: java.nio.ReadOnlyBufferException
      	at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350)
      	at org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660)
      	at org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728)
      	at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193)
      hadoopZeroCopyRead: ZeroCopyCursor#read failed error:
      ReadOnlyBufferException: java.nio.ReadOnlyBufferException
      	at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350)
      	at org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660)
      	at org.apache.hadoop.cF0301 08:25:39.334376 21960 hdfs-lzo-text-scanner.cc:582] 744cd34df9463fba:610c3d1000000003] Check failed: stream_->eof() 
      *** Check failure stack trace: ***
      

      This was testing with a slightly old impala-lzo commit: https://github.com/cloudera/impala-lzo/commit/36d19a8a4559ec6703e5879684b0d341c75c0202

      I'll attach logs and more info soon.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            tarmstrong Tim Armstrong
            tarmstrong Tim Armstrong
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment