Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Not A Bug
-
2.3.4
-
None
-
None
Description
We found that some class cannot be found after the regionserver has been running for a period of time(more than a month). And more than half of the machines in our cluster have this problem. Some are still running normally, but /rs-status cannot be opened，and others have various problems(such as rit, replication failed, abort failed)
Some exceptions are as follows:
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl at org.apache.hadoop.hbase.regionserver.RSStatusServlet.doGet(RSStatusServlet.java:49) at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) at org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:112) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) at org.apache.hadoop.hbase.http.SecurityHeadersFilter.doFilter(SecurityHeadersFilter.java:66) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) at org.apache.hadoop.hbase.http.ClickjackingPreventionFilter.doFilter(ClickjackingPreventionFilter.java:52) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) at org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1495)
2022-08-24 19:22:52,536 ERROR [RS_CLOSE_REGION-regionserver/xxxx:26020-1] regionserver.HRegionServer: ***** ABORTING region server 10.x.x.x,26020,1659357427101: Replay of WAL required. Forcing server shutdown *****
org.apache.hadoop.hbase.DroppedSnapshotException: region: xxxx:xxxx,xxxxxx,1661298619920.943104cbcf4a74db9896fd4abd051411.
at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2906)
at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2575)
at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2547)
at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2538)
at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1652)
at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1591)
at org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler.process(UnassignRegionHandler.java:114)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/regionserver/querymatcher/DeleteTracker$DeleteResult
at org.apache.hadoop.hbase.regionserver.querymatcher.ScanDeleteTracker.isDeleted(ScanDeleteTracker.java:108)
at org.apache.hadoop.hbase.regionserver.querymatcher.ScanQueryMatcher.checkDeleted(ScanQueryMatcher.java:209)
at org.apache.hadoop.hbase.regionserver.querymatcher.MinorCompactionScanQueryMatcher.match(MinorCompactionScanQueryMatcher.java:54)
at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:627)
at org.apache.hadoop.hbase.regionserver.StoreFlusher.performFlush(StoreFlusher.java:127)
at org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:69)
at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:1067)
at org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:2442)
at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2842)
... 10 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.regionserver.querymatcher.DeleteTracker$DeleteResult
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 19 more