Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
ghx-label-8
Description
I got a NoClassDefFoundError when launching catalogd deployed by the RPM package
I0411 13:38:14.642594 113789 init.cc:602] Default AES cipher mode for spill-to-disk: AES-GCM
I0411 13:38:14.746702 113789 init.cc:440] Using Java weigher sizeof
I0411 13:38:15.480756 113789 webserver.cc:395] Starting webserver on 0.0.0.0:25020
I0411 13:38:15.480811 113789 webserver.cc:411] Document root: /opt/impala
I0411 13:38:15.481134 113789 webserver.cc:548] Webserver started
I0411 13:38:15.609732 113789 jni-util.cc:302] java.lang.NoClassDefFoundError: org.apache.hadoop.fs.FileSystem
at org.apache.hadoop.hive.conf.valcoersion.JavaIOTmpdirVariableCoercion.<clinit>(JavaIOTmpdirVariableCoercion.java:37)
at org.apache.hadoop.hive.conf.SystemVariables.<clinit>(SystemVariables.java:37)
at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<init>(HiveConf.java:5736)
at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<init>(HiveConf.java:5717)
at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:536)
at org.apache.hadoop.hive.conf.HiveConf.<clinit>(HiveConf.java:256)
at org.apache.impala.service.JniCatalog.<clinit>(JniCatalog.java:135)
I0411 13:38:15.609787 113789 status.cc:129] NoClassDefFoundError: org.apache.hadoop.fs.FileSystem
@ 0x10a02f4
@ 0x1c21104
@ 0x1036151
@ 0xfd433d
@ 0xf5a43e
@ 0xe5b774
@ 0x7f81d2bf66a3
@ 0xf05c2e
F0411 13:38:15.611681 113789 catalog.cc:86] NoClassDefFoundError: org.apache.hadoop.fs.FileSystem
. Impalad exiting.
Minidump with no thread info available.
Wrote minidump to /var/log/impala-minidumps/catalogd/86355b1b-4d2c-4195-61ac0d9e-68059041.dmp
No more info in catalogd.ERROR:
Log file created at: 2024/04/11 13:38:14
Running on machine: ccycloud-3.quanlong.root.comops.site
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0411 13:38:14.627285 113789 logging.cc:256] stderr will be logged to this file.
log4j:WARN No appenders could be found for logger (org.apache.hadoop.conf.Configuration).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
F0411 13:38:15.611681 113789 catalog.cc:86] NoClassDefFoundError: org.apache.hadoop.fs.FileSystem
. Impalad exiting.
*** Check failure stack trace: ***
@ 0x391054d
@ 0x3912484
@ 0x390ff2c
@ 0x39129a9
@ 0x103625b
@ 0xfd433d
@ 0xf5a43e
@ 0xe5b774
@ 0x7f81d2bf66a3
@ 0xf05c2e
Minidump with no thread info available.
Wrote minidump to /var/log/impala-minidumps/catalogd/86355b1b-4d2c-4195-61ac0d9e-68059041.dmp
It seems an issue of the CLASSPATH that the jar of hadoop-common is missing. But actually in my case, it's due to the process doesn't have permission to read the core-site.xml file. Due to missing log4j.properties, the exception is not shown in the log, which makes it pretty hard to debug.
Note that the exception happens before GlogAppender is installed. So logs from log4j are not redirected to glog yet.
After adding the missing log4j.properties file, the exception is shown, explaning the NoClassDefFoundError. It's due to a failure in initializing a static field of FileSystem.
Log file created at: 2024/04/12 02:27:41
Running on machine: ccycloud-3.quanlong.root.comops.site
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0412 02:27:41.413901 190384 logging.cc:256] stderr will be logged to this file.
24/04/12 02:27:42 ERROR conf.Configuration: error parsing conf core-site.xml
java.io.FileNotFoundException: /opt/impala/conf/core-site.xml (Permission denied)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at java.io.FileInputStream.<init>(FileInputStream.java:93)
at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:3023)
at org.apache.hadoop.conf.Configuration.getStreamReader(Configuration.java:3119)
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3077)
at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:3050)
at org.apache.hadoop.conf.Configuration.loadProps(Configuration.java:2928)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2910)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:1264)
at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1318)
at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1543)
at org.apache.hadoop.fs.FileSystem$Cache.<init>(FileSystem.java:3495)
at org.apache.hadoop.fs.FileSystem.<clinit>(FileSystem.java:194)
24/04/12 02:27:42 INFO util.JvmPauseMonitor: Starting JVM pause monitor
24/04/12 02:27:42 INFO conf.HiveConf: Found configuration file file:/opt/impala/conf/hive-site.xml
F0412 02:27:42.422082 190384 catalog.cc:86] NoClassDefFoundError: org.apache.hadoop.fs.FileSystem
. Impalad exiting.
*** Check failure stack trace: ***
@ 0x391054d
@ 0x3912484
@ 0x390ff2c
@ 0x39129a9
@ 0x103625b
@ 0xfd433d
@ 0xf5a43e
@ 0xe5b774
@ 0x7fb2a83cd6a3
@ 0xf05c2e
Minidump with no thread info available.
Wrote minidump to /var/log/impala-minidumps/catalogd/1c092b19-c5db-442a-dbbfff82-47299b9e.dmp