[HBASE-15686] Add override mechanism for the exempt classes when dynamically loading table coprocessor - ASF JIRA

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 1.0.1
Fix Version/s: 1.4.0, 0.98.20, 2.0.0
Component/s: Coprocessors
Labels:
None

Hadoop Flags:

Reviewed
Release Note:

Hide
New coprocessor table descriptor attribute, hbase.coprocessor.classloader.included.classes, is added.
User can specify class name prefixes (semicolon separated) which should be loaded by CoprocessorClassLoader through this attribute using the following syntax:
{code}
hbase> alter 't1', 'coprocessor'=>'hdfs:///foo.jar|com.foo.FooRegionObserver|1001|arg1=1,arg2=2'
{code}

Show
New coprocessor table descriptor attribute, hbase.coprocessor.classloader.included.classes, is added. User can specify class name prefixes (semicolon separated) which should be loaded by CoprocessorClassLoader through this attribute using the following syntax: {code} hbase> alter 't1', 'coprocessor'=>' hdfs:///foo.jar |com.foo.FooRegionObserver|1001|arg1=1,arg2=2' {code}

Description

As part of Hadoop's Timeline Service v.2 (~~YARN-2928~~), we're adding a table coprocessor (~~YARN-4062~~). However, we're finding that the coprocessor cannot be loaded dynamically. A relevant snippet for the exception:

java.io.IOException: Class org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor cannot be loaded
    at org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1329)
    at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1269)
    at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:398)
    at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:42436)
    at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
    at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
    at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
    at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Class org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor cannot be loaded
    at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.testTableCoprocessorAttrs(RegionCoprocessorHost.java:324)
    at org.apache.hadoop.hbase.master.HMaster.checkClassLoading(HMaster.java:1483)
    at org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1327)
    ... 8 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at org.apache.hadoop.hbase.util.CoprocessorClassLoader.loadClass(CoprocessorClassLoader.java:275)
    at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.testTableCoprocessorAttrs(RegionCoprocessorHost.java:322)
    ... 10 more

We tracked it down to the fact that CoprocessorClassLoader regarding all hadoop classes as exempt from loading from the coprocessor jar. Since our coprocessor sits in the coprocessor jar, and yet the loading of this class is delegated to the parent which does not have this jar, the classloading fails.

What would be nice is the ability to exclude certain classes from the exempt classes so that they can be loaded via table coprocessor classloader. See hadoop's ApplicationClassLoader for a similar feature.

Is there any other way to load this coprocessor at the table scope?