Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-6073

No response from expr-codegen-test/expr-test

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Cannot Reproduce
    • Impala 2.9.0
    • None
    • Backend
    • None
    • ubuntu 14.04.5 LTS
    • ghx-label-4

    Description

      I checked out HEAD of Impala repo. There is no any code modification on my local repo.
      After full build, ./build/debug/exprs/expr-test and ./build/debug/exprs/expr-codegen-test seem to be hung. The other gtests in be are working appropriately. Some threads are waiting and the other threads are on sleep.

      I guess this issue is similar to the deadlock: https://issues.apache.org/jira/browse/HDFS-11851

      Please let me know if you have any workaround.

      Here are stack traces on expr-codegen-test using GDB:

      jinchulkim@ubuntu:~/workspace/Impala/be$ gdb ./build/debug/exprs/expr-codegen-test
      ...
      (gdb) info thread
        Id   Target Id         Frame
        16   Thread 0x7fffe20ae700 (LWP 23374) "expr-codegen-te" pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
        15   Thread 0x7fffe21af700 (LWP 23373) "expr-codegen-te" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
        14   Thread 0x7fffe22b0700 (LWP 23372) "expr-codegen-te" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
        13   Thread 0x7fffe23b1700 (LWP 23371) "expr-codegen-te" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
        12   Thread 0x7fffe24b2700 (LWP 23370) "expr-codegen-te" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
        11   Thread 0x7fffe2c95700 (LWP 23369) "expr-codegen-te" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
        10   Thread 0x7fffe2d96700 (LWP 23368) "expr-codegen-te" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
        9    Thread 0x7fffe2e97700 (LWP 23367) "expr-codegen-te" pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
        8    Thread 0x7fffe915a700 (LWP 23366) "expr-codegen-te" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
        7    Thread 0x7fffe925b700 (LWP 23365) "expr-codegen-te" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
        6    Thread 0x7fffe935c700 (LWP 23364) "expr-codegen-te" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
        5    Thread 0x7fffe945d700 (LWP 23363) "expr-codegen-te" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
        4    Thread 0x7fffee536700 (LWP 23362) "expr-codegen-te" 0x00007ffff037bb9d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
        3    Thread 0x7fffeed37700 (LWP 23361) "expr-codegen-te" 0x00007ffff037bb9d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
        2    Thread 0x7fffef538700 (LWP 23356) "expr-codegen-te" 0x00007ffff0067dfd in nanosleep () at ../sysdeps/unix/syscall-template.S:81
      * 1    Thread 0x7fffef5408c0 (LWP 23150) "expr-codegen-te" __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
      (gdb) where
      #0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
      #1  0x00007ffff0376649 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2  0x00007ffff0376470 in __GI___pthread_mutex_lock (mutex=0x7ffff4080600 <jvmMutex>) at ../nptl/pthread_mutex_lock.c:79
      #3  0x00007ffff3e7b666 in mutexLock (m=<optimized out>) at /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/os/posix/mutexes.c:28
      #4  0x00007ffff3e73a11 in setTLSExceptionStrings (rootCause=0x0, stackTrace=0x0) at /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:581
      #5  0x00007ffff3e73393 in printExceptionAndFreeV (env=0x307c1d8, exc=0x301adc0, noPrintFlags=<optimized out>, fmt=0x7ffff3e7bf6e "loadFileSystems", ap=0x7fffffffaab0) at /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c:183
      #6  0x00007ffff3e735ef in printExceptionAndFree (env=<optimized out>, exc=<optimized out>, noPrintFlags=<optimized out>, fmt=<optimized out>) at /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c:213
      #7  0x00007ffff3e74880 in getGlobalJNIEnv () at /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:463
      #8  getJNIEnv () at /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:528
      #9  0x00007ffff7845a1c in impala::JniUtil::Init () at /home/jinchulkim/workspace/Impala/be/src/util/jni-util.cc:105
      #10 0x00007ffff7e3c221 in impala::InitCommonRuntime (argc=1, argv=0x7fffffffb1e8, init_jvm=true, test_mode=impala::TestInfo::BE_TEST) at /home/jinchulkim/workspace/Impala/be/src/common/init.cc:236
      #11 0x00000000008333fc in main (argc=1, argv=0x7fffffffb1e8) at /home/jinchulkim/workspace/Impala/be/src/exprs/expr-codegen-test.cc:361
      (gdb) thr 2
      [Switching to thread 2 (Thread 0x7fffef538700 (LWP 23356))]
      #0  0x00007ffff0067dfd in nanosleep () at ../sysdeps/unix/syscall-template.S:81
      81      ../sysdeps/unix/syscall-template.S: No such file or directory.
      (gdb) where
      #0  0x00007ffff0067dfd in nanosleep () at ../sysdeps/unix/syscall-template.S:81
      #1  0x00007ffff0067c94 in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:137
      #2  0x00007ffff7e3b304 in LogMaintenanceThread () at /home/jinchulkim/workspace/Impala/be/src/common/init.cc:113
      #3  0x00007ffff56e4a8e in boost::detail::function::void_function_invoker0<void (*)(), void>::invoke (function_ptr=...) at /home/jinchulkim/workspace/Impala/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:112
      #4  0x00007ffff7b22f22 in boost::function0<void>::operator() (this=0x7fffef537ca0) at /home/jinchulkim/workspace/Impala/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
      #5  0x00007ffff78c0ba3 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) (name=..., category=..., functor=..., thread_started=0x7fffffffaa60) at /home/jinchulkim/workspace/Impala/be/src/util/thread.cc:352
      #6  0x00007ffff78ca32c in boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0&, int) (this=0x2fb47c0, f=@0x2fb47b8: 0x7ffff78c0884 <impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*)>, a=...)
          at /home/jinchulkim/workspace/Impala/toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:457
      #7  0x00007ffff78ca26f in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > >::operator()() (this=0x2fb47b8)
          at /home/jinchulkim/workspace/Impala/toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20
      #8  0x00007ffff78ca232 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run()
          (this=0x2fb4600) at /home/jinchulkim/workspace/Impala/toolchain/boost-1.57.0-p3/include/boost/thread/detail/thread.hpp:116
      #9  0x000000000090c53a in thread_proxy ()
      #10 0x00007ffff0374184 in start_thread (arg=0x7fffef538700) at pthread_create.c:312
      #11 0x00007ffff00a137d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
      (gdb)
      

      Here are the relevant environment variables:

      jinchulkim@ubuntu:~/workspace/Impala/be$ echo $JAVA_HOME
      /usr/lib/jvm/java-7-openjdk-amd64
      
      jinchulkim@ubuntu:~/workspace/Impala/be$ echo $BOOST_LIBRARYDIR
      /usr/lib/x86_64-linux-gnu
      
      jinchulkim@ubuntu:~/workspace/Impala/be$ echo $LD_LIBRARY_PATH
      :/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server:/home/jinchulkim/workspace/Impala/toolchain/cdh_components/hadoop-2.6.0-cdh5.14.0-SNAPSHOT//lib/native:/home/jinchulkim/workspace/Impala/toolchain/snappy-1.1.4/lib:/home/jinchulkim/workspace/Impala/../Impala-lzo/build:/home/jinchulkim/workspace/Impala/toolchain/gcc-4.9.2/lib64:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server:/home/jinchulkim/workspace/Impala/toolchain/cdh_components/hadoop-2.6.0-cdh5.14.0-SNAPSHOT//lib/native:/home/jinchulkim/workspace/Impala/toolchain/snappy-1.1.4/lib:/home/jinchulkim/workspace/Impala/../Impala-lzo/build:/home/jinchulkim/workspace/Impala/toolchain/gcc-4.9.2/lib64
      

      Attachments

        Issue Links

          Activity

            People

              philip Philip Martin
              jinchul Jin Chul Kim
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: