Description
This issue came up while integrating HCatalog with our environment. HCatalog jars are added to the pig command-line with -Dpig.additional.jars but fails (exception below). When added to the pig classpath the error goes away.
We identified the issue as deserialization using the root class loader, not the context class loader set when the thread is created. This causes HCatSchema which is serialized into the context to fail deserialization in the thread.
2012-02-14 21:55:53,936 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 6017: java.io.IOException: Deserialization error: org.apache.hcatalog.data.schema.HCatSchema at org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java:55) at org.apache.pig.impl.util.UDFContext.deserialize(UDFContext.java:181) at org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil.setupUDFContext(MapRedUtil.java:159) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:229) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:186) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:811) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:771) at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378) at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigJobControl.mainLoopAction(PigJobControl.java:144) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigJobControl.run(PigJobControl.java:121) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.ClassNotFoundException: org.apache.hcatalog.data.schema.HCatSchema at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:247) at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:603) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1574) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1495) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1731) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350) at java.util.Hashtable.readObject(Hashtable.java:859) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350) at java.util.HashMap.readObject(HashMap.java:1030) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350) at org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java:53) ... 15 more
Attachments
Attachments
Issue Links
- relates to
-
PIG-2576 Change in behavior for UDFContext.getUDFContext().getJobConf() in front-end
-
- Closed
-