[KUDU-905] tsan: RowsetMetadata::CommitUpdate() data race with ToProtobuf() - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: Feature Complete
Fix Version/s: None
Component/s: tablet
Labels:
None

Target Version/s:

Public beta
Code Review:
http://gerrit.sjc.cloudera.com:8080/#/c/7280/

Description

Saw this data race come up in a build at http://sandbox.jenkins.cloudera.com/job/kudu-gerrit/9158/BUILD_TYPE=TSAN,label=kudu-gerrit-slaves/testReport/junit/(root)/MultiThreadedTabletTest_3/DoTestAllAtOnce/

Very hard to reproduce.

Regression

MultiThreadedTabletTest/3.DoTestAllAtOnce (from MultiThreadedTabletTest_3)

Failing for the past 1 build (Since Failed#9158 )
Took 0 ms.
add description
Error Message

WARNING: ThreadSanitizer: data race (pid=27743)  Read of size 8 at 0x7d08000140c8 by thread T19 (mutexes: write M302061, write M296486, write M296489):
Stacktrace


WARNING: ThreadSanitizer: data race (pid=27743)  Read of size 8 at 0x7d08000140c8 by thread T19 (mutexes: write M302061, write M296486, write M296489):
    #0 kudu::BlockId::CopyToPB(kudu::BlockIdPB*) const /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/fs/block_id.cc:31:14 (libkudu_fs.so+0x0000000611dd)
    #1 kudu::tablet::RowSetMetadata::ToProtobuf(kudu::tablet::RowSetDataPB*) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/rowset_metadata.cc:88:5 (libtablet.so+0x0000002c0f22)
    #2 kudu::tablet::TabletMetadata::ToSuperBlockUnlocked(kudu::tablet::TabletSuperBlockPB*, std::vector<std::tr1::shared_ptr<kudu::tablet::RowSetMetadata>, std::allocator<std::tr1::shared_ptr<kudu::tablet::RowSetMetadata> > > const&) const /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet_metadata.cc:384:5 (libtablet.so+0x0000002b873d)
    #3 kudu::tablet::TabletMetadata::Flush() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet_metadata.cc:285:5 (libtablet.so+0x0000002b5eef)
    #4 kudu::tablet::TabletMetadata::UpdateAndFlush(std::tr1::unordered_set<long, std::tr1::hash<long>, std::equal_to<long>, std::allocator<long> > const&, std::vector<std::tr1::shared_ptr<kudu::tablet::RowSetMetadata>, std::allocator<std::tr1::shared_ptr<kudu::tablet::RowSetMetadata> > > const&, long) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet_metadata.cc:213:10 (libtablet.so+0x0000002b7d6e)
    #5 kudu::tablet::Tablet::FlushMetadata(std::vector<std::tr1::shared_ptr<kudu::tablet::RowSet>, std::allocator<std::tr1::shared_ptr<kudu::tablet::RowSet> > > const&, std::vector<std::tr1::shared_ptr<kudu::tablet::RowSetMetadata>, std::allocator<std::tr1::shared_ptr<kudu::tablet::RowSetMetadata> > > const&, long) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet.cc:1076:10 (libtablet.so+0x0000001d716e)
    #6 kudu::tablet::Tablet::DoCompactionOrFlush(kudu::tablet::RowSetsInCompaction const&, long) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet.cc:1259:3 (libtablet.so+0x0000001d2e09)
    #7 kudu::tablet::Tablet::FlushInternal(kudu::tablet::RowSetsInCompaction const&, std::tr1::shared_ptr<kudu::tablet::MemRowSet> const&) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet.cc:617:3 (libtablet.so+0x0000001d1e3a)
    #8 kudu::tablet::Tablet::FlushUnlocked() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet.cc:560:10 (libtablet.so+0x0000001d1874)
    #9 kudu::tablet::Tablet::Flush() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet.cc:542:10 (libtablet.so+0x0000001d16a0)
    #10 kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >::FlushThread(int) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/mt-tablet-test.cc:256:9 (mt-tablet-test+0x00000010e2ad)
    #11 boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>::operator()(kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >*, int) const /usr/include/boost/bind/mem_fn_template.hpp:162:29 (mt-tablet-test+0x000000111474)
    #12 void boost::_bi::list2<boost::_bi::value<kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >*>, boost::_bi::value<int> >::operator()<boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:306:9 (mt-tablet-test+0x0000001113da)
    #13 boost::_bi::bind_t<void, boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>, boost::_bi::list2<boost::_bi::value<kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >*>, boost::_bi::value<int> > >::operator()() /usr/include/boost/bind/bind_template.hpp:20:27 (mt-tablet-test+0x000000111363)
    #14 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>, boost::_bi::list2<boost::_bi::value<kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >*>, boost::_bi::value<int> > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153:11 (mt-tablet-test+0x000000111171)
    #15 boost::function0<void>::operator()() const /usr/include/boost/function/function_template.hpp:1012:12 (libkrpc.so+0x000000091af1)
    #16 kudu::Thread::SuperviseThread(void*) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/thread.cc:478:3 (libkudu_util.so+0x0000001ea0b9)

  Previous write of size 8 at 0x7d08000140c8 by thread T25 (mutexes: write M312998, write M313055, write M312908):
    #0 bool UpdateReturnCopy<std::tr1::unordered_map<int, kudu::BlockId, std::tr1::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, kudu::BlockId> > > >(std::tr1::unordered_map<int, kudu::BlockId, std::tr1::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, kudu::BlockId> > >*, std::tr1::unordered_map<int, kudu::BlockId, std::tr1::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, kudu::BlockId> > >::value_type::first_type const&, std::tr1::unordered_map<int, kudu::BlockId, std::tr1::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, kudu::BlockId> > >::value_type::second_type const&, std::tr1::unordered_map<int, kudu::BlockId, std::tr1::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, kudu::BlockId> > >::value_type::first_type const&*) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/gutil/map-util.h:582:23 (libtablet.so+0x0000002c359c)
    #1 kudu::tablet::RowSetMetadata::CommitUpdate(kudu::tablet::RowSetMetadataUpdate const&) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/rowset_metadata.cc:185:11 (libtablet.so+0x0000002c19a0)
    #2 kudu::tablet::DiskRowSet::MajorCompactDeltaStoresWithColumnIds(std::vector<int, std::allocator<int> > const&) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/diskrowset.cc:494:3 (libtablet.so+0x0000002666af)
    #3 kudu::tablet::DiskRowSet::MajorCompactDeltaStores() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/diskrowset.cc:477:10 (libtablet.so+0x000000266541)
    #4 kudu::tablet::Tablet::CompactWorstDeltas(kudu::tablet::RowSet::DeltaCompactionType) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet.cc:1582:5 (libtablet.so+0x0000001d581e)
    #5 kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >::CompactDeltas(kudu::tablet::RowSet::DeltaCompactionType) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/mt-tablet-test.cc:292:7 (mt-tablet-test+0x000000111b1d)
    #6 kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >::MajorCompactDeltasThread(int) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/mt-tablet-test.cc:286:5 (mt-tablet-test+0x00000010e7eb)
    #7 boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>::operator()(kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >*, int) const /usr/include/boost/bind/mem_fn_template.hpp:162:29 (mt-tablet-test+0x000000111474)
    #8 void boost::_bi::list2<boost::_bi::value<kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >*>, boost::_bi::value<int> >::operator()<boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:306:9 (mt-tablet-test+0x0000001113da)
    #9 boost::_bi::bind_t<void, boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>, boost::_bi::list2<boost::_bi::value<kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >*>, boost::_bi::value<int> > >::operator()() /usr/include/boost/bind/bind_template.hpp:20:27 (mt-tablet-test+0x000000111363)
    #10 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>, boost::_bi::list2<boost::_bi::value<kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >*>, boost::_bi::value<int> > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153:11 (mt-tablet-test+0x000000111171)
    #11 boost::function0<void>::operator()() const /usr/include/boost/function/function_template.hpp:1012:12 (libkrpc.so+0x000000091af1)
    #12 kudu::Thread::SuperviseThread(void*) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/thread.cc:478:3 (libkudu_util.so+0x0000001ea0b9)

  Location is heap block of size 24 at 0x7d08000140c0 allocated by thread T19:
    #0 operator new(unsigned long) <null> (mt-tablet-test+0x0000000823b8)
    #1 __gnu_cxx::new_allocator<std::tr1::__detail::_Hash_node<std::pair<int const, kudu::BlockId>, false> >::allocate(unsigned long, void const*) /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/ext/new_allocator.h:89:27 (libtablet.so+0x00000023e0fd)
    #2 std::tr1::_Hashtable<int, std::pair<int const, kudu::BlockId>, std::allocator<std::pair<int const, kudu::BlockId> >, std::_Select1st<std::pair<int const, kudu::BlockId> >, std::equal_to<int>, std::tr1::hash<int>, std::tr1::__detail::_Mod_range_hashing, std::tr1::__detail::_Default_ranged_hash, std::tr1::__detail::_Prime_rehash_policy, false, false, true>::_M_allocate_node(std::pair<int const, kudu::BlockId> const&) /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/hashtable:479:20 (libtablet.so+0x00000023ddec)
    #3 std::tr1::_Hashtable<int, std::pair<int const, kudu::BlockId>, std::allocator<std::pair<int const, kudu::BlockId> >, std::_Select1st<std::pair<int const, kudu::BlockId> >, std::equal_to<int>, std::tr1::hash<int>, std::tr1::__detail::_Mod_range_hashing, std::tr1::__detail::_Default_ranged_hash, std::tr1::__detail::_Prime_rehash_policy, false, false, true>::_Hashtable(std::tr1::_Hashtable<int, std::pair<int const, kudu::BlockId>, std::allocator<std::pair<int const, kudu::BlockId> >, std::_Select1st<std::pair<int const, kudu::BlockId> >, std::equal_to<int>, std::tr1::hash<int>, std::tr1::__detail::_Mod_range_hashing, std::tr1::__detail::_Default_ranged_hash, std::tr1::__detail::_Prime_rehash_policy, false, false, true> const&) /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/hashtable:660:15 (libtablet.so+0x00000023dc7a)
    #4 std::tr1::_Hashtable<int, std::pair<int const, kudu::BlockId>, std::allocator<std::pair<int const, kudu::BlockId> >, std::_Select1st<std::pair<int const, kudu::BlockId> >, std::equal_to<int>, std::tr1::hash<int>, std::tr1::__detail::_Mod_range_hashing, std::tr1::__detail::_Default_ranged_hash, std::tr1::__detail::_Prime_rehash_policy, false, false, true>::operator=(std::tr1::_Hashtable<int, std::pair<int const, kudu::BlockId>, std::allocator<std::pair<int const, kudu::BlockId> >, std::_Select1st<std::pair<int const, kudu::BlockId> >, std::equal_to<int>, std::tr1::hash<int>, std::tr1::__detail::_Mod_range_hashing, std::tr1::__detail::_Default_ranged_hash, std::tr1::__detail::_Prime_rehash_policy, false, false, true> const&) /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/hashtable:711:18 (libtablet.so+0x0000002c433a)
    #5 std::tr1::__unordered_map<int, kudu::BlockId, std::tr1::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, kudu::BlockId> >, false>::operator=(std::tr1::__unordered_map<int, kudu::BlockId, std::tr1::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, kudu::BlockId> >, false> const&) /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/unordered_map:41:11 (libtablet.so+0x0000002c4300)
    #6 std::tr1::unordered_map<int, kudu::BlockId, std::tr1::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, kudu::BlockId> > >::operator=(std::tr1::unordered_map<int, kudu::BlockId, std::tr1::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, kudu::BlockId> > > const&) /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/unordered_map:188:11 (libtablet.so+0x0000002c2da0)
    #7 kudu::tablet::RowSetMetadata::SetColumnDataBlocks(std::tr1::unordered_map<int, kudu::BlockId, std::tr1::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, kudu::BlockId> > > const&) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/rowset_metadata.cc:124:21 (libtablet.so+0x0000002c1244)
    #8 kudu::tablet::DiskRowSetWriter::FinishAndReleaseBlocks(kudu::fs::ScopedWritableBlockCloser*) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/diskrowset.cc:202:3 (libtablet.so+0x00000026452c)
    #9 kudu::tablet::RollingDiskRowSetWriter::FinishCurrentWriter() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/diskrowset.cc:366:3 (libtablet.so+0x0000002654d3)
    #10 kudu::tablet::RollingDiskRowSetWriter::Finish() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/diskrowset.cc:401:3 (libtablet.so+0x000000265bcb)
    #11 kudu::tablet::Tablet::DoCompactionOrFlush(kudu::tablet::RowSetsInCompaction const&, long) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet.cc:1103:3 (libtablet.so+0x0000001d22c0)
    #12 kudu::tablet::Tablet::FlushInternal(kudu::tablet::RowSetsInCompaction const&, std::tr1::shared_ptr<kudu::tablet::MemRowSet> const&) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet.cc:617:3 (libtablet.so+0x0000001d1e3a)
    #13 kudu::tablet::Tablet::FlushUnlocked() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet.cc:560:10 (libtablet.so+0x0000001d1874)
    #14 kudu::tablet::Tablet::Flush() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet.cc:542:10 (libtablet.so+0x0000001d16a0)
    #15 kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >::FlushThread(int) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/mt-tablet-test.cc:256:9 (mt-tablet-test+0x00000010e2ad)
    #16 boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>::operator()(kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >*, int) const /usr/include/boost/bind/mem_fn_template.hpp:162:29 (mt-tablet-test+0x000000111474)
    #17 void boost::_bi::list2<boost::_bi::value<kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >*>, boost::_bi::value<int> >::operator()<boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:306:9 (mt-tablet-test+0x0000001113da)
    #18 boost::_bi::bind_t<void, boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>, boost::_bi::list2<boost::_bi::value<kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >*>, boost::_bi::value<int> > >::operator()() /usr/include/boost/bind/bind_template.hpp:20:27 (mt-tablet-test+0x000000111363)
    #19 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>, boost::_bi::list2<boost::_bi::value<kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >*>, boost::_bi::value<int> > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153:11 (mt-tablet-test+0x000000111171)
    #20 boost::function0<void>::operator()() const /usr/include/boost/function/function_template.hpp:1012:12 (libkrpc.so+0x000000091af1)
    #21 kudu::Thread::SuperviseThread(void*) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/thread.cc:478:3 (libkudu_util.so+0x0000001ea0b9)

  Mutex M302061 (0x7d4c00026cc8) created at:
    #0 pthread_mutex_init <null> (mt-tablet-test+0x0000000768d5)
    #1 boost::mutex::mutex() /usr/include/boost/thread/pthread/mutex.hpp:37:27 (libtablet.so+0x0000001db0cb)
    #2 kudu::tablet::MemRowSet::MemRowSet(long, kudu::Schema const&, kudu::log::LogAnchorRegistry*, std::tr1::shared_ptr<kudu::MemTracker> const&) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/memrowset.cc:70:12 (libtablet.so+0x00000027a20f)
    #3 kudu::tablet::Tablet::ReplaceMemRowSetUnlocked(kudu::tablet::RowSetsInCompaction*, std::tr1::shared_ptr<kudu::tablet::MemRowSet>*) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet.cc:575:37 (libtablet.so+0x0000001d1a80)
    #4 kudu::tablet::Tablet::FlushUnlocked() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet.cc:552:5 (libtablet.so+0x0000001d1832)
    #5 kudu::tablet::Tablet::Flush() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet.cc:542:10 (libtablet.so+0x0000001d16a0)
    #6 kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >::FlushThread(int) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/mt-tablet-test.cc:256:9 (mt-tablet-test+0x00000010e2ad)
    #7 boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>::operator()(kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >*, int) const /usr/include/boost/bind/mem_fn_template.hpp:162:29 (mt-tablet-test+0x000000111474)
    #8 void boost::_bi::list2<boost::_bi::value<kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >*>, boost::_bi::value<int> >::operator()<boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:306:9 (mt-tablet-test+0x0000001113da)
    #9 boost::_bi::bind_t<void, boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>, boost::_bi::list2<boost::_bi::value<kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >*>, boost::_bi::value<int> > >::operator()() /usr/include/boost/bind/bind_template.hpp:20:27 (mt-tablet-test+0x000000111363)
    #10 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>, boost::_bi::list2<boost::_bi::value<kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >*>, boost::_bi::value<int> > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153:11 (mt-tablet-test+0x000000111171)
    #11 boost::function0<void>::operator()() const /usr/include/boost/function/function_template.hpp:1012:12 (libkrpc.so+0x000000091af1)
    #12 kudu::Thread::SuperviseThread(void*) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/thread.cc:478:3 (libkudu_util.so+0x0000001ea0b9)

  Mutex M296486 (0x7d4400009b10) created at:
    #0 pthread_mutex_init <null> (mt-tablet-test+0x0000000768d5)
    #1 kudu::Mutex::Mutex() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/mutex.cc:28:8 (libkudu_util.so+0x0000001bdbd8)
    #2 kudu::tablet::TabletMetadata::TabletMetadata(kudu::FsManager*, std::string const&, std::string const&, kudu::Schema const&, std::string const&, std::string const&, kudu::tablet::TabletBootstrapStatePB const&) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet_metadata.cc:95:17 (libtablet.so+0x0000002b68de)
    #3 kudu::tablet::TabletMetadata::CreateNew(kudu::FsManager*, std::string const&, std::string const&, kudu::Schema const&, std::string const&, std::string const&, kudu::tablet::TabletBootstrapStatePB const&, scoped_refptr<kudu::tablet::TabletMetadata>*) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet_metadata.cc:50:41 (libtablet.so+0x0000002b5ca9)
    #4 kudu::tablet::TabletMetadata::LoadOrCreate(kudu::FsManager*, std::string const&, std::string const&, kudu::Schema const&, std::string const&, std::string const&, kudu::tablet::TabletBootstrapStatePB const&, scoped_refptr<kudu::tablet::TabletMetadata>*) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet_metadata.cc:87:12 (libtablet.so+0x0000002b66f8)
    #5 kudu::tablet::TabletHarness::Create(bool) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet-harness.h:59:5 (mt-tablet-test+0x0000000ecb3d)
    #6 kudu::tablet::KuduTabletTest::CreateTestTablet(std::string const&) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet-test-util.h:54:5 (mt-tablet-test+0x0000000ec5bc)
    #7 kudu::tablet::KuduTabletTest::SetUpTestTablet(std::string const&) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet-test-util.h:58:5 (mt-tablet-test+0x0000000ec404)
    #8 kudu::tablet::KuduTabletTest::SetUp() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet-test-util.h:45:5 (mt-tablet-test+0x0000000e6bdc)
    #9 kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >::SetUp() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/mt-tablet-test.cc:56:5 (mt-tablet-test+0x00000010cb11)
    #10 HandleSehExceptionsInMethodIfSupported<testing::Test, void> /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/thirdparty/gmock-1.7.0/gtest/src/gtest.cc:2078 (libgmock.so+0x00000003ff2a)
    #11 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/thirdparty/gmock-1.7.0/gtest/src/gtest.cc:2114 (libgmock.so+0x00000003ff2a)
    #12 main /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/test_main.cc:37:13 (libkudu_test_main.so+0x000000003b31)

  Mutex M296489 (0x7d4400009b0c) created at:
    #0 __tsan_atomic32_compare_exchange_strong <null> (mt-tablet-test+0x0000000ae428)
    #1 base::subtle::Acquire_CompareAndSwap(int volatile*, int, int) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/gutil/atomicops-internals-tsan.h:85:3 (libtablet.so+0x0000001e76f7)
    #2 base::SpinLock::Lock() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/gutil/spinlock.h:73:9 (libtablet.so+0x0000001e7670)
    #3 kudu::simple_spinlock::lock() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/locks.h:31:5 (libtablet.so+0x0000001e7639)
    #4 boost::lock_guard<kudu::simple_spinlock>::lock_guard(kudu::simple_spinlock&) /usr/include/boost/thread/locks.hpp:194:13 (libtablet.so+0x0000001df718)
    #5 kudu::tablet::TabletMetadata::Flush() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet_metadata.cc:276:33 (libtablet.so+0x0000002b5de6)
    #6 kudu::tablet::TabletMetadata::CreateNew(kudu::FsManager*, std::string const&, std::string const&, kudu::Schema const&, std::string const&, std::string const&, kudu::tablet::TabletBootstrapStatePB const&, scoped_refptr<kudu::tablet::TabletMetadata>*) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet_metadata.cc:57:3 (libtablet.so+0x0000002b5ccf)
    #7 kudu::tablet::TabletMetadata::LoadOrCreate(kudu::FsManager*, std::string const&, std::string const&, kudu::Schema const&, std::string const&, std::string const&, kudu::tablet::TabletBootstrapStatePB const&, scoped_refptr<kudu::tablet::TabletMetadata>*) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet_metadata.cc:87:12 (libtablet.so+0x0000002b66f8)
    #8 kudu::tablet::TabletHarness::Create(bool) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet-harness.h:59:5 (mt-tablet-test+0x0000000ecb3d)
    #9 kudu::tablet::KuduTabletTest::CreateTestTablet(std::string const&) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet-test-util.h:54:5 (mt-tablet-test+0x0000000ec5bc)
    #10 kudu::tablet::KuduTabletTest::SetUpTestTablet(std::string const&) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet-test-util.h:58:5 (mt-tablet-test+0x0000000ec404)
    #11 kudu::tablet::KuduTabletTest::SetUp() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet-test-util.h:45:5 (mt-tablet-test+0x0000000e6bdc)
    #12 kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >::SetUp() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/mt-tablet-test.cc:56:5 (mt-tablet-test+0x00000010cb11)
    #13 HandleSehExceptionsInMethodIfSupported<testing::Test, void> /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/thirdparty/gmock-1.7.0/gtest/src/gtest.cc:2078 (libgmock.so+0x00000003ff2a)
    #14 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/thirdparty/gmock-1.7.0/gtest/src/gtest.cc:2114 (libgmock.so+0x00000003ff2a)
    #15 main /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/test_main.cc:37:13 (libkudu_test_main.so+0x000000003b31)

  Mutex M312998 (0x7d2400041730) created at:
    #0 pthread_mutex_init <null> (mt-tablet-test+0x0000000768d5)
    #1 boost::mutex::mutex() /usr/include/boost/thread/pthread/mutex.hpp:37:27 (libtablet.so+0x0000001db0cb)
    #2 kudu::tablet::DiskRowSet::DiskRowSet(std::tr1::shared_ptr<kudu::tablet::RowSetMetadata> const&, kudu::log::LogAnchorRegistry*, std::tr1::shared_ptr<kudu::MemTracker> const&) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/diskrowset.cc:432:13 (libtablet.so+0x0000002662bc)
    #3 kudu::tablet::DiskRowSet::Open(std::tr1::shared_ptr<kudu::tablet::RowSetMetadata> const&, kudu::log::LogAnchorRegistry*, std::tr1::shared_ptr<kudu::tablet::DiskRowSet>*, std::tr1::shared_ptr<kudu::MemTracker> const&) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/diskrowset.cc:424:33 (libtablet.so+0x000000265f47)
    #4 kudu::tablet::Tablet::DoCompactionOrFlush(kudu::tablet::RowSetsInCompaction const&, long) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet.cc:1141:18 (libtablet.so+0x0000001d2501)
    #5 kudu::tablet::Tablet::FlushInternal(kudu::tablet::RowSetsInCompaction const&, std::tr1::shared_ptr<kudu::tablet::MemRowSet> const&) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet.cc:617:3 (libtablet.so+0x0000001d1e3a)
    #6 kudu::tablet::Tablet::FlushUnlocked() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet.cc:560:10 (libtablet.so+0x0000001d1874)
    #7 kudu::tablet::Tablet::Flush() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet.cc:542:10 (libtablet.so+0x0000001d16a0)
    #8 kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >::FlushThread(int) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/mt-tablet-test.cc:256:9 (mt-tablet-test+0x00000010e2ad)
    #9 boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>::operator()(kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >*, int) const /usr/include/boost/bind/mem_fn_template.hpp:162:29 (mt-tablet-test+0x000000111474)
    #10 void boost::_bi::list2<boost::_bi::value<kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >*>, boost::_bi::value<int> >::operator()<boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:306:9 (mt-tablet-test+0x0000001113da)
    #11 boost::_bi::bind_t<void, boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>, boost::_bi::list2<boost::_bi::value<kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >*>, boost::_bi::value<int> > >::operator()() /usr/include/boost/bind/bind_template.hpp:20:27 (mt-tablet-test+0x000000111363)
    #12 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>, boost::_bi::list2<boost::_bi::value<kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >*>, boost::_bi::value<int> > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153:11 (mt-tablet-test+0x000000111171)
    #13 boost::function0<void>::operator()() const /usr/include/boost/function/function_template.hpp:1012:12 (libkrpc.so+0x000000091af1)
    #14 kudu::Thread::SuperviseThread(void*) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/thread.cc:478:3 (libkudu_util.so+0x0000001ea0b9)

  Mutex M313055 (0x7d3000000f80) created at:
    #0 pthread_mutex_init <null> (mt-tablet-test+0x0000000768d5)
    #1 kudu::Mutex::Mutex() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/mutex.cc:28:8 (libkudu_util.so+0x0000001bdbd8)
    #2 kudu::tablet::DeltaTracker::DeltaTracker(std::tr1::shared_ptr<kudu::tablet::RowSetMetadata> const&, unsigned int, kudu::log::LogAnchorRegistry*, std::tr1::shared_ptr<kudu::MemTracker> const&) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/delta_tracker.cc:32:15 (libtablet.so+0x0000002f5f86)
    #3 kudu::tablet::DiskRowSet::Open() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/diskrowset.cc:449:28 (libtablet.so+0x000000266105)
    #4 kudu::tablet::DiskRowSet::Open(std::tr1::shared_ptr<kudu::tablet::RowSetMetadata> const&, kudu::log::LogAnchorRegistry*, std::tr1::shared_ptr<kudu::tablet::DiskRowSet>*, std::tr1::shared_ptr<kudu::MemTracker> const&) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/diskrowset.cc:426:3 (libtablet.so+0x000000265f69)
    #5 kudu::tablet::Tablet::DoCompactionOrFlush(kudu::tablet::RowSetsInCompaction const&, long) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet.cc:1141:18 (libtablet.so+0x0000001d2501)
    #6 kudu::tablet::Tablet::FlushInternal(kudu::tablet::RowSetsInCompaction const&, std::tr1::shared_ptr<kudu::tablet::MemRowSet> const&) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet.cc:617:3 (libtablet.so+0x0000001d1e3a)
    #7 kudu::tablet::Tablet::FlushUnlocked() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet.cc:560:10 (libtablet.so+0x0000001d1874)
    #8 kudu::tablet::Tablet::Flush() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet.cc:542:10 (libtablet.so+0x0000001d16a0)
    #9 kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >::FlushThread(int) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/mt-tablet-test.cc:256:9 (mt-tablet-test+0x00000010e2ad)
    #10 boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>::operator()(kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >*, int) const /usr/include/boost/bind/mem_fn_template.hpp:162:29 (mt-tablet-test+0x000000111474)
    #11 void boost::_bi::list2<boost::_bi::value<kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >*>, boost::_bi::value<int> >::operator()<boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:306:9 (mt-tablet-test+0x0000001113da)
    #12 boost::_bi::bind_t<void, boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>, boost::_bi::list2<boost::_bi::value<kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >*>, boost::_bi::value<int> > >::operator()() /usr/include/boost/bind/bind_template.hpp:20:27 (mt-tablet-test+0x000000111363)
    #13 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>, boost::_bi::list2<boost::_bi::value<kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >*>, boost::_bi::value<int> > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153:11 (mt-tablet-test+0x000000111171)
    #14 boost::function0<void>::operator()() const /usr/include/boost/function/function_template.hpp:1012:12 (libkrpc.so+0x000000091af1)
    #15 kudu::Thread::SuperviseThread(void*) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/thread.cc:478:3 (libkudu_util.so+0x0000001ea0b9)

  Mutex M312908 (0x7d2400047ee4) created at:
    #0 __tsan_atomic32_compare_exchange_strong <null> (mt-tablet-test+0x0000000ae428)
    #1 base::subtle::Acquire_CompareAndSwap(int volatile*, int, int) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/gutil/atomicops-internals-tsan.h:85:3 (libtablet.so+0x0000001e76f7)
    #2 base::SpinLock::Lock() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/gutil/spinlock.h:73:9 (libtablet.so+0x0000001e7670)
    #3 kudu::simple_spinlock::lock() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/locks.h:31:5 (libtablet.so+0x0000001e7639)
    #4 boost::lock_guard<kudu::simple_spinlock>::lock_guard(kudu::simple_spinlock&) /usr/include/boost/thread/locks.hpp:194:13 (libtablet.so+0x0000001df718)
    #5 kudu::tablet::RowSetMetadata::CommitUndoDeltaDataBlock(kudu::BlockId const&) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/rowset_metadata.cc:136:31 (libtablet.so+0x0000002c1318)
    #6 kudu::tablet::RollingDiskRowSetWriter::FinishCurrentWriter() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/diskrowset.cc:374:5 (libtablet.so+0x0000002655b0)
    #7 kudu::tablet::RollingDiskRowSetWriter::Finish() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/diskrowset.cc:401:3 (libtablet.so+0x000000265bcb)
    #8 kudu::tablet::Tablet::DoCompactionOrFlush(kudu::tablet::RowSetsInCompaction const&, long) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet.cc:1103:3 (libtablet.so+0x0000001d22c0)
    #9 kudu::tablet::Tablet::FlushInternal(kudu::tablet::RowSetsInCompaction const&, std::tr1::shared_ptr<kudu::tablet::MemRowSet> const&) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet.cc:617:3 (libtablet.so+0x0000001d1e3a)
    #10 kudu::tablet::Tablet::FlushUnlocked() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet.cc:560:10 (libtablet.so+0x0000001d1874)
    #11 kudu::tablet::Tablet::Flush() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/tablet.cc:542:10 (libtablet.so+0x0000001d16a0)
    #12 kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >::FlushThread(int) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/mt-tablet-test.cc:256:9 (mt-tablet-test+0x00000010e2ad)
    #13 boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>::operator()(kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >*, int) const /usr/include/boost/bind/mem_fn_template.hpp:162:29 (mt-tablet-test+0x000000111474)
    #14 void boost::_bi::list2<boost::_bi::value<kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >*>, boost::_bi::value<int> >::operator()<boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>&, boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:306:9 (mt-tablet-test+0x0000001113da)
    #15 boost::_bi::bind_t<void, boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>, boost::_bi::list2<boost::_bi::value<kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >*>, boost::_bi::value<int> > >::operator()() /usr/include/boost/bind/bind_template.hpp:20:27 (mt-tablet-test+0x000000111363)
    #16 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf1<void, kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >, int>, boost::_bi::list2<boost::_bi::value<kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >*>, boost::_bi::value<int> > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153:11 (mt-tablet-test+0x000000111171)
    #17 boost::function0<void>::operator()() const /usr/include/boost/function/function_template.hpp:1012:12 (libkrpc.so+0x000000091af1)
    #18 kudu::Thread::SuperviseThread(void*) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/thread.cc:478:3 (libkudu_util.so+0x0000001ea0b9)

  Thread T19 'test0-27936' (tid=27936, running) created by main thread at:
    #0 pthread_create <null> (mt-tablet-test+0x00000008c3d3)
    #1 kudu::Thread::StartThread(std::string const&, std::string const&, boost::function<void ()()> const&, scoped_refptr<kudu::Thread>*) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/thread.cc:406:13 (libkudu_util.so+0x0000001e9b76)
    #2 kudu::Status kudu::Thread::Create<void (kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >::*)(int), kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >*, int>(std::string const&, std::string const&, void (kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >::* const&)(int), kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >* const&, int const&, scoped_refptr<kudu::Thread>*) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/thread.h:132:12 (mt-tablet-test+0x000000110d6a)
    #3 void kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >::StartThreads<void (kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >::*)(int)>(int, void (kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >::* const&)(int)) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/mt-tablet-test.cc:363:7 (mt-tablet-test+0x00000010da09)
    #4 kudu::tablet::MultiThreadedTabletTest_DoTestAllAtOnce_Test<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >::TestBody() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/mt-tablet-test.cc:401:3 (mt-tablet-test+0x00000010d093)
    #5 HandleSehExceptionsInMethodIfSupported<testing::Test, void> /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/thirdparty/gmock-1.7.0/gtest/src/gtest.cc:2078 (libgmock.so+0x00000003ff2a)
    #6 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/thirdparty/gmock-1.7.0/gtest/src/gtest.cc:2114 (libgmock.so+0x00000003ff2a)
    #7 main /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/test_main.cc:37:13 (libkudu_test_main.so+0x000000003b31)

  Thread T25 'test0-27941' (tid=27941, running) created by main thread at:
    #0 pthread_create <null> (mt-tablet-test+0x00000008c3d3)
    #1 kudu::Thread::StartThread(std::string const&, std::string const&, boost::function<void ()()> const&, scoped_refptr<kudu::Thread>*) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/thread.cc:406:13 (libkudu_util.so+0x0000001e9b76)
    #2 kudu::Status kudu::Thread::Create<void (kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >::*)(int), kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >*, int>(std::string const&, std::string const&, void (kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >::* const&)(int), kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >* const&, int const&, scoped_refptr<kudu::Thread>*) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/thread.h:132:12 (mt-tablet-test+0x000000110d6a)
    #3 void kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >::StartThreads<void (kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >::*)(int)>(int, void (kudu::tablet::MultiThreadedTabletTest<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >::* const&)(int)) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/mt-tablet-test.cc:363:7 (mt-tablet-test+0x00000010da09)
    #4 kudu::tablet::MultiThreadedTabletTest_DoTestAllAtOnce_Test<kudu::tablet::IntKeyTestSetup<(kudu::DataType)5> >::TestBody() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/tablet/mt-tablet-test.cc:406:3 (mt-tablet-test+0x00000010d174)
    #5 HandleSehExceptionsInMethodIfSupported<testing::Test, void> /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/thirdparty/gmock-1.7.0/gtest/src/gtest.cc:2078 (libgmock.so+0x00000003ff2a)
    #6 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/thirdparty/gmock-1.7.0/gtest/src/gtest.cc:2114 (libgmock.so+0x00000003ff2a)
    #7 main /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/test_main.cc:37:13 (libkudu_test_main.so+0x000000003b31)

Attachments

Activity

People

Assignee:: Mike Percy

Reporter:: Mike Percy

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: 30/Jul/15 15:50

Updated:: 19/Aug/15 20:18

Resolved:: 19/Aug/15 20:18