Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4800

apparent crash during stress run

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Duplicate
    • Impala 2.9.0
    • None
    • Backend

    Description

      During a stress run over the weekend, there was a failure with printed backtrace.

      [...skipped...]
      #5  <signal handler called>
      #6  0x0000000001351d0b in base::subtle::Acquire_Load (ptr=0x18) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/be/src/gutil/atomicops-internals-x86.h:320
      #7  0x0000000001352124 in Load (this=0x0) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/be/src/common/atomic.h:81
      #8  impala::RuntimeProfile::HighWaterMarkCounter::current_value (this=0x0) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/be/src/util/runtime-profile-counters.h:111
      #9  0x0000000001352212 in impala::MemTracker::consumption (this=0x19c6e8f20) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/be/src/runtime/mem-tracker.h:286
      #10 0x00000000019d2b49 in boost::_mfi::cmf0<long, impala::MemTracker>::call<impala::MemTracker* const> (this=0x7fd46815ec98, u=@0x7fd46815eca8) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/toolchain/boost-1.57.0-p1/include/boost/bind/mem_fn_template.hpp:105
      #11 0x00000000019d2991 in boost::_mfi::cmf0<long, impala::MemTracker>::operator()<impala::MemTracker*> (this=0x7fd46815ec98, u=@0x7fd46815eca8) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/toolchain/boost-1.57.0-p1/include/boost/bind/mem_fn_template.hpp:115
      #12 0x00000000019d2599 in boost::_bi::list1<boost::_bi::value<impala::MemTracker*> >::operator()<long, boost::_mfi::cmf0<long, impala::MemTracker>, boost::_bi::list0> (this=0x7fd46815eca8, f=..., a=...) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/toolchain/boost-1.57.0-p1/include/boost/bind/bind.hpp:243
      #13 0x00000000019d232d in boost::_bi::bind_t<long, boost::_mfi::cmf0<long, impala::MemTracker>, boost::_bi::list1<boost::_bi::value<impala::MemTracker*> > >::operator() (this=0x7fd46815ec98) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/toolchain/boost-1.57.0-p1/include/boost/bind/bind_template.hpp:20
      #14 0x00000000019d210f in boost::detail::function::function_obj_invoker0<boost::_bi::bind_t<long, boost::_mfi::cmf0<long, impala::MemTracker>, boost::_bi::list1<boost::_bi::value<impala::MemTracker*> > >, long>::invoke (function_obj_ptr=...) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/toolchain/boost-1.57.0-p1/include/boost/function/function_template.hpp:132
      #15 0x00000000015a01c4 in boost::function0<long>::operator() (this=0x7fd46815ec90) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/toolchain/boost-1.57.0-p1/include/boost/function/function_template.hpp:767
      #16 0x000000000159fe53 in impala::RuntimeProfile::TimeSeriesCounter::AddSample (this=0x7fd46815ec80, ms_elapsed=500) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/be/src/util/runtime-profile-counters.h:342
      #17 0x000000000159f879 in impala::PeriodicCounterUpdater::UpdateLoop (this=0x3f3da00) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/be/src/util/periodic-counter-updater.cc:185
      #18 0x00000000015a9acf in boost::_mfi::mf0<void, impala::PeriodicCounterUpdater>::operator() (this=0x4e7a1b8, p=0x3f3da00) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/toolchain/boost-1.57.0-p1/include/boost/bind/mem_fn_template.hpp:49
      #19 0x00000000015a9a32 in boost::_bi::list1<boost::_bi::value<impala::PeriodicCounterUpdater*> >::operator()<boost::_mfi::mf0<void, impala::PeriodicCounterUpdater>, boost::_bi::list0> (this=0x4e7a1c8, f=..., a=...) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/toolchain/boost-1.57.0-p1/include/boost/bind/bind.hpp:253
      #20 0x00000000015a99dd in boost::_bi::bind_t<void, boost::_mfi::mf0<void, impala::PeriodicCounterUpdater>, boost::_bi::list1<boost::_bi::value<impala::PeriodicCounterUpdater*> > >::operator() (this=0x4e7a1b8) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/toolchain/boost-1.57.0-p1/include/boost/bind/bind_template.hpp:20
      #21 0x00000000015a99a0 in boost::detail::thread_data<boost::_bi::bind_t<void, boost::_mfi::mf0<void, impala::PeriodicCounterUpdater>, boost::_bi::list1<boost::_bi::value<impala::PeriodicCounterUpdater*> > > >::run (this=0x4e7a000) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/toolchain/boost-1.57.0-p1/include/boost/thread/detail/thread.hpp:116
      #22 0x0000000001a3cdaa in thread_proxy ()
      #23 0x00000039c68079d1 in start_thread () from /lib64/libpthread.so.0
      #24 0x00000039c64e88fd in clone () from /lib64/libc.so.6
      

      Unfortunately a different run blew away the logs/minidumps before they could be collected. Also, a subsequent run passed on the same Git hash (IMPALA-4775: minor adjustments to python test infra logging). I am re-running the stress test repeatedly to see if I can reproduce this problem.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              mikeb Michael Brown
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: