Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
2.0.0-beta-1
-
None
-
beginner
Description
In the past, the hbase-indexer runs multiple RegionServers per JVM. In old days, they had their own cut-down "RegionServer". In 2.0.0, we made it so they could run an actual RegionServer but with services disabled. The latter has an issue if you run more than one instance per JVM and it is NOT a minihbasecluster instance. It fails with:
1:09:13.371 PM ERROR HRegionServer Failed init org.apache.hadoop.metrics2.MetricsException: Metrics source RegionServer,sub=Server already exists! at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152) at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229) at org.apache.hadoop.hbase.metrics.BaseSourceImpl.<init>(BaseSourceImpl.java:115) at org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceImpl.<init>(MetricsRegionServerSourceImpl.java:101) at org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceImpl.<init>(MetricsRegionServerSourceImpl.java:93) at org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactoryImpl.createServer(MetricsRegionServerSourceFactoryImpl.java:69) at org.apache.hadoop.hbase.regionserver.MetricsRegionServer.<init>(MetricsRegionServer.java:56) at org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1519) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:954) at com.ngdata.sep.impl.SepConsumer$1.run(SepConsumer.java:203)
If you look in [10:26 AM] Wolfgang Hoschek: DefaultMetricsSystem code (found by whoschek), you'll see this:
synchronized ObjectName newObjectName(String name) { try { if (mBeanNames.map.containsKey(name) && !miniClusterMode) { throw new MetricsException(name +" already exists!"); } return new ObjectName(mBeanNames.uniqueName(name)); } catch (Exception e) { throw new MetricsException(e); } }
i.e. if we are in a mini cluster context, we will not fail registering the second bean instance.
If you look in master startup in HMasterCommandLine, you will see:
// If 'local', defer to LocalHBaseCluster instance. Starts master // and regionserver both in the one JVM. if (LocalHBaseCluster.isLocal(conf)) { DefaultMetricsSystem.setMiniClusterMode(true); ....
... will ensure we don't get the above exception in minihbasecluster context.
So, the idea here is to make it so being able to run more than one RS per JVM is cleaner than doing the above hack. It needs to be a config too.... a config. which says don't fail startup if second mbean registration just because two RS in the one context (A later issue will be the accounting of metrics per RS... If more than one RS, then we should make a unique mbean per RS in the JVM).