Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Invalid
-
2.3.0
-
None
-
None
Description
After running mvn clean package, I tried to execute one of the spark example program JavaDirectKafkaWordCount.java but throws following exeception.
[cloud-user@server-2 examples]$ run-example streaming.JavaDirectKafkaWordCount 192.168.0.4:9092 msu 2018-04-25 09:39:22 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-04-25 09:39:22 INFO SparkContext:54 - Running Spark version 2.3.0 2018-04-25 09:39:22 INFO SparkContext:54 - Submitted application: JavaDirectKafkaWordCount 2018-04-25 09:39:22 INFO SecurityManager:54 - Changing view acls to: cloud-user 2018-04-25 09:39:22 INFO SecurityManager:54 - Changing modify acls to: cloud-user 2018-04-25 09:39:22 INFO SecurityManager:54 - Changing view acls groups to: 2018-04-25 09:39:22 INFO SecurityManager:54 - Changing modify acls groups to: 2018-04-25 09:39:22 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(cloud-user); groups with view permissions: Set(); users with modify permissions: Set(cloud-user); groups with modify permissions: Set() 2018-04-25 09:39:23 INFO Utils:54 - Successfully started service 'sparkDriver' on port 59333. 2018-04-25 09:39:23 INFO SparkEnv:54 - Registering MapOutputTracker 2018-04-25 09:39:23 INFO SparkEnv:54 - Registering BlockManagerMaster 2018-04-25 09:39:23 INFO BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 2018-04-25 09:39:23 INFO BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up 2018-04-25 09:39:23 INFO DiskBlockManager:54 - Created local directory at /tmp/blockmgr-6fc11fc1-f638-42ea-a9df-dc01fb81b7b6 2018-04-25 09:39:23 INFO MemoryStore:54 - MemoryStore started with capacity 366.3 MB 2018-04-25 09:39:23 INFO SparkEnv:54 - Registering OutputCommitCoordinator 2018-04-25 09:39:23 INFO log:192 - Logging initialized @1825ms 2018-04-25 09:39:23 INFO Server:346 - jetty-9.3.z-SNAPSHOT 2018-04-25 09:39:23 INFO Server:414 - Started @1900ms 2018-04-25 09:39:23 INFO AbstractConnector:278 - Started ServerConnector@6813a331{HTTP/1.1,[http/1.1]}{0.0.0.0:4040} 2018-04-25 09:39:23 INFO Utils:54 - Successfully started service 'SparkUI' on port 4040. 2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4f7c0be3{/jobs,null,AVAILABLE,@Spark} 2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4cfbaf4{/jobs/json,null,AVAILABLE,@Spark} 2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@58faa93b{/jobs/job,null,AVAILABLE,@Spark} 2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@127d7908{/jobs/job/json,null,AVAILABLE,@Spark} 2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6b9c69a9{/stages,null,AVAILABLE,@Spark} 2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6622a690{/stages/json,null,AVAILABLE,@Spark} 2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@30b9eadd{/stages/stage,null,AVAILABLE,@Spark} 2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3249a1ce{/stages/stage/json,null,AVAILABLE,@Spark} 2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4dd94a58{/stages/pool,null,AVAILABLE,@Spark} 2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2f4919b0{/stages/pool/json,null,AVAILABLE,@Spark} 2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@a8a8b75{/storage,null,AVAILABLE,@Spark} 2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@75b21c3b{/storage/json,null,AVAILABLE,@Spark} 2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@72be135f{/storage/rdd,null,AVAILABLE,@Spark} 2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@155d1021{/storage/rdd/json,null,AVAILABLE,@Spark} 2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4bd2f0dc{/environment,null,AVAILABLE,@Spark} 2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2e647e59{/environment/json,null,AVAILABLE,@Spark} 2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2c42b421{/executors,null,AVAILABLE,@Spark} 2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@51e37590{/executors/json,null,AVAILABLE,@Spark} 2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@deb3b60{/executors/threadDump,null,AVAILABLE,@Spark} 2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@701a32{/executors/threadDump/json,null,AVAILABLE,@Spark} 2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@39aa45a1{/static,null,AVAILABLE,@Spark} 2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@294bdeb4{/,null,AVAILABLE,@Spark} 2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5300f14a{/api,null,AVAILABLE,@Spark} 2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@54acff7d{/jobs/job/kill,null,AVAILABLE,@Spark} 2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7bc9e6ab{/stages/stage/kill,null,AVAILABLE,@Spark} 2018-04-25 09:39:23 INFO SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://server-2:4040 2018-04-25 09:39:23 INFO SparkContext:54 - Added JAR file:///home/cloud-user/spark-2.3.0-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.0.jar at spark://server-2:59333/jars/spark-examples_2.11-2.3.0.jar with timestamp 1524663563504 2018-04-25 09:39:23 INFO SparkContext:54 - Added JAR file:///home/cloud-user/spark-2.3.0-bin-hadoop2.7/examples/jars/scopt_2.11-3.7.0.jar at spark://server-2:59333/jars/scopt_2.11-3.7.0.jar with timestamp 1524663563505 2018-04-25 09:39:23 INFO Executor:54 - Starting executor ID driver on host localhost 2018-04-25 09:39:23 INFO Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 56363. 2018-04-25 09:39:23 INFO NettyBlockTransferService:54 - Server created on server-2:56363 2018-04-25 09:39:23 INFO BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 2018-04-25 09:39:23 INFO BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, server-2, 56363, None) 2018-04-25 09:39:23 INFO BlockManagerMasterEndpoint:54 - Registering block manager server-2:56363 with 366.3 MB RAM, BlockManagerId(driver, server-2, 56363, None) 2018-04-25 09:39:23 INFO BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, server-2, 56363, None) 2018-04-25 09:39:23 INFO BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, server-2, 56363, None) 2018-04-25 09:39:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4e2916c3{/metrics/json,null,AVAILABLE,@Spark} Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/streaming/kafka010/LocationStrategies at org.apache.spark.examples.streaming.JavaDirectKafkaWordCount.main(JavaDirectKafkaWordCount.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.ClassNotFoundException: org.apache.spark.streaming.kafka010.LocationStrategies at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 11 more 2018-04-25 09:39:23 INFO SparkContext:54 - Invoking stop() from shutdown hook 2018-04-25 09:39:23 INFO AbstractConnector:318 - Stopped Spark@6813a331{HTTP/1.1,[http/1.1]}{0.0.0.0:4040} 2018-04-25 09:39:23 INFO SparkUI:54 - Stopped Spark web UI at http://server-2:4040 2018-04-25 09:39:23 INFO MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped! 2018-04-25 09:39:23 INFO MemoryStore:54 - MemoryStore cleared 2018-04-25 09:39:23 INFO BlockManager:54 - BlockManager stopped 2018-04-25 09:39:23 INFO BlockManagerMaster:54 - BlockManagerMaster stopped 2018-04-25 09:39:23 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped! 2018-04-25 09:39:23 INFO SparkContext:54 - Successfully stopped SparkContext 2018-04-25 09:39:23 INFO ShutdownHookManager:54 - Shutdown hook called 2018-04-25 09:39:23 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-edc94694-ab74-4b66-9ef2-10d28b3f5359 2018-04-25 09:39:23 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-d57ca1de-b096-4036-ad4b-ed97295443c4