Details
-
Bug
-
Status: Open
-
Critical
-
Resolution: Unresolved
-
Impala 3.4.2
-
None
-
ghx-label-9
Description
I tried deploying the RPM package of Impala-3.4.2 (commit 8e9c5a5) on CentOS 7.9 and found launching catalogd failed by the following error (in catalogd.INFO):
Wrote minidump to /var/log/impala-minidumps/catalogd/5e3c8819-0593-4943-555addbc-665470ad.dmp # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x0000000002baf14c, pid=156082, tid=0x00007fec0dce59c0 # # JRE version: Java(TM) SE Runtime Environment (8.0_141-b15) (build 1.8.0_141-b15) # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.141-b15 mixed mode linux-amd64 compressed oops) # Problematic frame: # C [catalogd+0x27af14c] llvm::SCEVAddRecExpr::getNumIterationsInRange(llvm::ConstantRange const&, llvm::ScalarEvolution&) const+0x73c # # Core dump written. Default location: /opt/impala/core or core.156082 # # An error report file with more information is saved as: # /tmp/hs_err_pid156082.log # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # The crash happened outside the Java Virtual Machine in native code. # See problematic frame for where to report the bug. #
There are other logs in catalogd.ERROR
Log file created at: 2024/04/08 04:49:28 Running on machine: ccycloud-1.quanlong.root.comops.site Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg E0408 04:49:28.979386 158187 logging.cc:146] stderr will be logged to this file. Wrote minidump to /var/log/impala-minidumps/catalogd/6c3f550c-be96-4a5b-61171aac-0de15155.dmp could not find method getRootCauseMessage from class (null) with signature (Ljava/lang/Throwable;)Ljava/lang/String; could not find method getStackTrace from class (null) with signature (Ljava/lang/Throwable;)Ljava/lang/String; FileSystem: loadFileSystems failed error: (unable to get root cause for java.lang.NoClassDefFoundError) (unable to get stack trace for java.lang.NoClassDefFoundError)
Resolving the minidump shows me the following stacktrace:
(gdb) bt #0 0x0000000002baf14c in ?? () #1 0x0000000002baee24 in getJNIEnv () #2 0x0000000002bacb71 in hdfsBuilderConnect () #3 0x00000000012e6ae2 in impala::JniUtil::InitLibhdfs() () #4 0x00000000012e7897 in impala::JniUtil::Init() () #5 0x0000000000be9297 in impala::InitCommonRuntime(int, char**, bool, impala::TestInfo::Mode) () #6 0x0000000000bb604a in CatalogdMain(int, char**) () #7 0x0000000000b33f97 in main ()
It indicates something wrong in initializing the JVM. Here are the env vars:
Environment Variables: JAVA_HOME=/usr/java/jdk1.8.0_141 CLASSPATH=/opt/impala/conf:/opt/impala/jar/* PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/root/bin LD_LIBRARY_PATH=/opt/impala/lib/:/usr/java/jdk1.8.0_141/jre/lib/amd64/server:/usr/java/jdk1.8.0_141/jre/lib/amd64 SHELL=/bin/bash
We use wildcard "*" in the classpath which seems to be the cause. The issue was resolved after using explicit paths in the classpath. Here are what I changed in bin/impala-env.sh:
#export CLASSPATH="/opt/impala/conf:/opt/impala/jar/*" CLASSPATH=/opt/impala/conf for jar in /opt/impala/jar/*.jar; do CLASSPATH="$CLASSPATH:$jar" done export CLASSPATH