Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-12915

double-free memory issue in unifiedbetests if it's built in dynamically linked mode

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • Impala 4.4.0
    • Backend
    • None
    • ghx-label-3

    Description

      I have a pretty simple change that just adds one new metric in common/thrift/metrics.json: https://github.com/stiga-huang/impala/commit/3972aed68428b3f307e823f49a2700c1a98aa3b3
      It breaks the clang-tidy job in the step of running "unifiedbetests --gtest_list_tests".

      The job builds Impala using the -so option which means dynamically link executables (default is static). I'm able to reproduce the issue locally. Using gdb, I can see the process crashed by a memory issue:

      (gdb) bt
      #0  0x0000000002ff8d6b in tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned int, int) ()
      #1  0x0000000002ff93c5 in tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned int) ()
      #2  0x00007fffef06db1f in std::_Function_base::_Base_manager<boost::algorithm::detail::is_any_ofF<char> >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) ()
         from /home/quanlong/workspace/Impala/be/build/release/kudu_util/libkudu_util.so
      #3  0x00007fffef06d7e4 in std::_Function_handler<bool (char), boost::algorithm::detail::is_any_ofF<char> >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) ()
         from /home/quanlong/workspace/Impala/be/build/release/kudu_util/libkudu_util.so
      #4  0x000000000109fc07 in std::_Function_base::~_Function_base() ()
      #5  0x00007fffed7a8605 in __cxa_finalize () from /lib/x86_64-linux-gnu/libc.so.6
      #6  0x00007fffeef995a3 in __do_global_dtors_aux () from /home/quanlong/workspace/Impala/be/build/release/kudu_util/libkudu_util.so
      #7  0x00007fffffff03c0 in ?? ()

      boroknagyz found that upgrading gtest to 1.14 resolves the issue. The current gtest version is 1.6. I tried building with ASAN and also build the underlying gtest-1.6 with ASAN. The run then shows the memory issue:

      ==1778==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000098530 at pc 0x00000123a9a9 bp 0x7ffeeaf47d60 sp 0x7ffeeaf47d58
      READ of size 8 at 0x603000098530 thread T0
          #0 0x123a9a8 in testing::internal::String::~String() /home/quanlong/workspace/Impala/toolchain/toolchain-packages-gcc10.4.0/gtest-1.6.0/include/gtest/internal/gtest-string.h:218:24
          #1 0x530fc2f in void std::_Destroy_aux<false>::__destroy<testing::internal::String*>(testing::internal::String*, testing::internal::String*) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_construct.h:108:6
          #2 0x52fdcdd in std::vector<testing::internal::String, std::allocator<testing::internal::String> >::~vector() /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_vector.h:677:2
          #3 0x7f4ca763a604 in __cxa_finalize /build/glibc-CVJwZb/glibc-2.27/stdlib/cxa_finalize.c:83
          #4 0x7f4ca93a8cb2 in __do_global_dtors_aux (/home/quanlong/workspace/Impala/be/src/kudu/util/libkudu_test_util.so+0x6ecb2)
      
      0x603000098530 is located 0 bytes inside of 32-byte region [0x603000098530,0x603000098550)
      freed by thread T0 here:
          #0 0x1233dcf in operator delete(void*) (/home/quanlong/workspace/Impala/be/build/debug/service/unifiedbetests+0x1233dcf)
          #1 0x5310da3 in std::_Vector_base<testing::internal::String, std::allocator<testing::internal::String> >::~_Vector_base() /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_vector.h:332:2
          #2 0x7f4ca763a030 in __run_exit_handlers /build/glibc-CVJwZb/glibc-2.27/stdlib/exit.c:108
      
      previously allocated by thread T0 here:
          #0 0x123305f in operator new(unsigned long) (/home/quanlong/workspace/Impala/be/build/debug/service/unifiedbetests+0x123305f)
          #1 0x530fdd0 in std::vector<testing::internal::String, std::allocator<testing::internal::String> >::_M_realloc_insert(__gnu_cxx::__normal_iterator<testing::internal::String*, std::vector<testing::internal::String, std::allocator<testing::internal::String> > >, testing::internal::String const&) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/vector.tcc:440:33
          #2 0x52fc053 in void testing::internal::InitGoogleTestImpl<char>(int*, char**) /media/quanlong/hdd-backup/native-toolchain/source/gtest/gtest-1.6.0/src/gtest.cc:4868:13
          #3 0x1236435 in main /home/quanlong/workspace/Impala/be/src/service/unified-betest-main.cc:40:3
          #4 0x7f4ca7618c86 in __libc_start_main /build/glibc-CVJwZb/glibc-2.27/csu/../csu/libc-start.c:310

      Note that I have to remove the definition of "THttpTransport::CRLF_LEN" in be/src/transport/THttpTransport.cpp to avoid an ODR violation issue.

      libkudu_test_util.so wants to delete a vector that is already deleted by gtest.
      https://github.com/google/googletest/blob/release-1.6.0/include/gtest/internal/gtest-port.h#L1057
       
      In the static build, we are fine since there are no libkudu_test_util.so or libkudu_util.so.

      Attachments

        Issue Links

          Activity

            People

              stigahuang Quanlong Huang
              stigahuang Quanlong Huang
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: