Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-17324

Don't relocate org.bouncycastle in shaded client jars

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      When downstream apps depend on hadoop-client-api, hadoop-client-runtime and hadoop-client-minicluster, it seems the MiniYARNCluster could have issue because org.apache.hadoop.shaded.org.bouncycastle.operator.OperatorCreationException is not in any of the above jars. 

      Error:  Caused by: sbt.ForkMain$ForkError: java.lang.ClassNotFoundException: org.apache.hadoop.shaded.org.bouncycastle.operator.OperatorCreationException
Error:  	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
Error:  	at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
Error:  	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
Error:  	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
Error:  	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:862)
Error:  	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
Error:  	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1296)
Error:  	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:339)
Error:  	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
Error:  	at org.apache.hadoop.yarn.server.MiniYARNCluster.initResourceManager(MiniYARNCluster.java:353)
Error:  	at org.apache.hadoop.yarn.server.MiniYARNCluster.access$200(MiniYARNCluster.java:127)
Error:  	at org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceInit(MiniYARNCluster.java:488)
Error:  	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
Error:  	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:109)
Error:  	at org.apache.hadoop.yarn.server.MiniYARNCluster.serviceInit(MiniYARNCluster.java:321)
Error:  	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
Error:  	at org.apache.spark.deploy.yarn.BaseYarnClusterSuite.beforeAll(BaseYarnClusterSuite.scala:94)
Error:  	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
Error:  	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
Error:  	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
Error:  	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:61)
Error:  	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:318)
Error:  	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:513)
Error:  	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:413)
Error:  	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
Error:  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Error:  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Error:  	at java.lang.Thread.run(Thread.java:748)

      Attachments

        Issue Links

          blocks

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-29250 Upgrade to Hadoop 3.3.1

          • Major - Major loss of function.
          • Resolved
          duplicates

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-17327 NPE when starting MiniYARNCluster from hadoop-client-minicluster

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          is broken by

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-15832 Upgrade BouncyCastle to 1.60

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. HBASE-28295 Few tests are failing due to NCDFE: org/bouncycastle/operator/OperatorCreationException

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-17373 hadoop-client-integration-tests doesn't work when building with skipShade

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #2411

          Activity

            People

              csun Chao Sun
              csun Chao Sun
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 4h 50m
                  4h 50m