Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3200 Replace BufferedBlockMgr with new buffer pool
  3. IMPALA-5130

MemTracker::EnableReservationReporting() is not thread-safe

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • Impala 2.9.0
    • Impala 2.9.0
    • Backend
    • None
    • ghx-label-6

    Description

      MemTracker::EnableReservationReporting() can run concurrently with MemTracker::LogUsage(). There is no synchronisation between the two.

      I saw a crash with this stack on a development branch, which I believe is caused by this:

      (gdb) bt
      #0  0x00000032836328e5 in raise () from sysroot/lib64/libc.so.6
      #1  0x00000032836340c5 in abort () from sysroot/lib64/libc.so.6
      #2  0x00007f2f0b8f7c55 in os::abort(bool) () from sysroot/opt/toolchain/sun-jdk-64bit-1.7.0.75/jre/lib/amd64/server/libjvm.so
      #3  0x00007f2f0ba79cd7 in VMError::report_and_die() () from sysroot/opt/toolchain/sun-jdk-64bit-1.7.0.75/jre/lib/amd64/server/libjvm.so
      #4  0x00007f2f0b8fcb6f in JVM_handle_linux_signal () from sysroot/opt/toolchain/sun-jdk-64bit-1.7.0.75/jre/lib/amd64/server/libjvm.so
      #5  <signal handler called>
      #6  0x00007f2f0a75ee50 in vtable for __cxxabiv1::__si_class_type_info () from sysroot/data/jenkins/workspace/impala-private-build-and-test/Impala-Toolchain/gcc-4.9.2/lib64/libstdc++.so.6
      #7  0x00000000013ac8de in impala::MemTracker::LogUsage (this=0x5be3c80, prefix=...) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/mem-tracker.cc:253
      #8  0x00000000013acdab in impala::MemTracker::LogUsage (prefix=..., trackers=...) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/mem-tracker.cc:278
      #9  0x00000000013acb86 in impala::MemTracker::LogUsage (this=0xb03a6e0, prefix=...) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/mem-tracker.cc:269
      #10 0x00000000013acdab in impala::MemTracker::LogUsage (prefix=..., trackers=...) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/mem-tracker.cc:278
      #11 0x00000000013acb86 in impala::MemTracker::LogUsage (this=0x157626e0, prefix=...) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/mem-tracker.cc:269
      #12 0x00000000013acdab in impala::MemTracker::LogUsage (prefix=..., trackers=...) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/mem-tracker.cc:278
      #13 0x00000000013acb86 in impala::MemTracker::LogUsage (this=0x9612260, prefix=...) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/mem-tracker.cc:269
      #14 0x00000000013acdab in impala::MemTracker::LogUsage (prefix=..., trackers=...) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/mem-tracker.cc:278
      #15 0x00000000013acb86 in impala::MemTracker::LogUsage (this=0x5be3440, prefix=...) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/mem-tracker.cc:269
      #16 0x00000000016b175d in impala::ExecNode::Prepare (this=0x14d55a20, state=0xb056000) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/exec/exec-node.cc:174
      #17 0x00000000017fd22f in impala::SortNode::Prepare (this=0x14d55a20, state=0xb056000) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/exec/sort-node.cc:49
      #18 0x0000000001a6a48a in impala::PlanFragmentExecutor::PrepareInternal (this=0x22dea2d0, qs=0x1c569c00, tdesc_tbl=..., fragment_ctx=..., instance_ctx=...)
          at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/plan-fragment-executor.cc:214
      #19 0x0000000001a68bcd in impala::PlanFragmentExecutor::Prepare (this=0x22dea2d0, query_state=0x1c569c00, desc_tbl=..., fragment_ctx=..., instance_ctx=...)
          at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/plan-fragment-executor.cc:99
      #20 0x0000000001a65b49 in impala::FragmentInstanceState::Exec (this=0x22dea000) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/fragment-instance-state.cc:64
      #21 0x0000000001a71189 in impala::QueryExecMgr::ExecFInstance (this=0x8add2a0, fis=0x22dea000) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/query-exec-mgr.cc:110
      #22 0x0000000001a73fb2 in boost::_mfi::mf1<void, impala::QueryExecMgr, impala::FragmentInstanceState*>::operator() (this=0x1ef7fec0, p=0x8add2a0, a1=0x22dea000)
          at /data/jenkins/workspace/impala-private-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/mem_fn_template.hpp:165
      #23 0x0000000001a73e3b in boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, boost::_bi::value<impala::FragmentInstanceState*> >::operator()<boost::_mfi::mf1<void, impala::QueryExecMgr, impala::FragmentInstanceState*>, boost::_bi::list0> (
          this=0x1ef7fed0, f=..., a=...) at /data/jenkins/workspace/impala-private-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind.hpp:313
      #24 0x0000000001a7396f in boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::QueryExecMgr, impala::FragmentInstanceState*>, boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, boost::_bi::value<impala::FragmentInstanceState*> > >::operator() (
          this=0x1ef7fec0) at /data/jenkins/workspace/impala-private-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind_template.hpp:20
      #25 0x0000000001a7357a in boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::QueryExecMgr, impala::FragmentInstanceState*>, boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, boost::_bi::value<impala::FragmentInstanceState*> > >, void>::invoke (function_obj_ptr=...) at /data/jenkins/workspace/impala-private-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/function/function_template.hpp:153
      #26 0x00000000013923d6 in boost::function0<void>::operator() (this=0x7f2e34e55c40) at /data/jenkins/workspace/impala-private-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/function/function_template.hpp:767
      #27 0x000000000162e6d1 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) (name=..., category=..., functor=..., thread_started=0x7f2e10b079a0)
          at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/util/thread.cc:325
      #28 0x00000000016370ac in boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0&, int) (this=0x171b6dc0, 
          f=@0x171b6db8: 0x162e40c <impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*)>, a=...)
          at /data/jenkins/workspace/impala-private-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind.hpp:457
      #29 0x0000000001636fef in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > >::operator()() (this=0x171b6db8) at /data/jenkins/workspace/impala-private-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind_template.hpp:20
      #30 0x0000000001636fb2 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run() (this=0x171b6c00) at /data/jenkins/workspace/impala-private-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/thread/detail/thread.hpp:116
      #31 0x0000000001aee87a in thread_proxy ()
      #32 0x0000003283a07851 in start_thread () from sysroot/lib64/libpthread.so.0
      #33 0x00000032836e894d in clone () from sysroot/lib64/libc.so.6
      

      on this line:

      int64_t reservation_limit = reservation_counters_->reservation_limit->value();
      

      This method is not currently used in query execution, but we need to fix this before switching on the buffer pool for query execution.

      Attachments

        Activity

          People

            tarmstrong Tim Armstrong
            tarmstrong Tim Armstrong
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: