Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25075 Build and test Spark against Scala 2.13
  3. SPARK-35532

Ensure mllib and kafka-0-10 module can maven test independently in Scala 2.13

Rank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersConvert to IssueLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.0
    • 3.2.0
    • Tests
    • None

    Description

      Execute

       

      mvn test -pl mllib -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive -Pscala-2.13
      

       

      6 java tests error:

       

      [ERROR] Errors: 
      [ERROR]   JavaStreamingLogisticRegressionSuite.javaAPI:78 » TestFailed 20005 was not les...
      [ERROR]   JavaStreamingKMeansSuite.javaAPI:78 » TestFailed 20040 was not less than 20000...
      [ERROR]   JavaPrefixSpanSuite.runPrefixSpan:45 » NoClassDefFound scala/collection/parall...
      [ERROR]   JavaPrefixSpanSuite.runPrefixSpanSaveLoad:67 » NoClassDefFound scala/collectio...
      [ERROR]   JavaStreamingLinearRegressionSuite.javaAPI:77 » TestFailed 20014 was not less ...
      [ERROR]   JavaStatisticsSuite.streamingTest:112 » TestFailed 20043 was not less than 200...
      

       

      The error message is 

       

      Exception in thread "streaming-start" java.lang.NoClassDefFoundError: scala/collection/parallel/TaskSupport
      	at org.apache.spark.streaming.util.FileBasedWriteAheadLog.<init>(FileBasedWriteAheadLog.scala:62)
      	at org.apache.spark.streaming.util.WriteAheadLogUtils$.$anonfun$createLog$2(WriteAheadLogUtils.scala:125)
      	at scala.Option.getOrElse(Option.scala:201)
      	at org.apache.spark.streaming.util.WriteAheadLogUtils$.createLog(WriteAheadLogUtils.scala:123)
      	at org.apache.spark.streaming.util.WriteAheadLogUtils$.createLogForDriver(WriteAheadLogUtils.scala:81)
      	at org.apache.spark.streaming.scheduler.ReceivedBlockTracker.$anonfun$createWriteAheadLog$1(ReceivedBlockTracker.scala:265)
      	at scala.Option.map(Option.scala:242)
      	at org.apache.spark.streaming.scheduler.ReceivedBlockTracker.createWriteAheadLog(ReceivedBlockTracker.scala:263)
      	at org.apache.spark.streaming.scheduler.ReceivedBlockTracker.<init>(ReceivedBlockTracker.scala:76)
      	at org.apache.spark.streaming.scheduler.ReceiverTracker.<init>(ReceiverTracker.scala:114)
      	at org.apache.spark.streaming.scheduler.JobScheduler.start(JobScheduler.scala:86)
      	at org.apache.spark.streaming.StreamingContext.$anonfun$start$1(StreamingContext.scala:590)
      	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
      	at org.apache.spark.util.ThreadUtils$$anon$2.run(ThreadUtils.scala:218)
      Caused by: java.lang.ClassNotFoundException: scala.collection.parallel.TaskSupport
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
      	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
      	... 14 more
      

       

      Execute

       

      mvn test -pl external/kafka-0-10 -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive -Pscala-2.13
      

       

      1 java test error:

       

      [ERROR] Errors: 
      [ERROR] org.apache.spark.streaming.kafka010.JavaDirectKafkaStreamSuite.testKafkaStream
      [ERROR]   Run 1: JavaDirectKafkaStreamSuite.testKafkaStream:170 expected:<[topic1-1, topic1-2, topic2-1, topic1-3, topic2-2, topic2-3]> but was:<[]>
      [ERROR]   Run 2: JavaDirectKafkaStreamSuite.tearDown:57 » NoClassDefFound scala/collection/para...
      

       

      The error message is:

      [ERROR] Tests run: 2, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 20.772 s <<< FAILURE! - in org.apache.spark.streaming.kafka010.JavaDirectKafkaStreamSuite
      [ERROR] org.apache.spark.streaming.kafka010.JavaDirectKafkaStreamSuite.testKafkaStream  Time elapsed: 20.765 s  <<< FAILURE!
      java.lang.AssertionError: expected:<[topic1-1, topic1-2, topic2-1, topic1-3, topic2-2, topic2-3]> but was:<[]>
      	at org.apache.spark.streaming.kafka010.JavaDirectKafkaStreamSuite.testKafkaStream(JavaDirectKafkaStreamSuite.java:170)
      
      
      [ERROR] org.apache.spark.streaming.kafka010.JavaDirectKafkaStreamSuite.testKafkaStream  Time elapsed: 20.766 s  <<< ERROR!
      java.lang.NoClassDefFoundError: scala/collection/parallel/immutable/ParVector
      	at org.apache.spark.streaming.kafka010.JavaDirectKafkaStreamSuite.tearDown(JavaDirectKafkaStreamSuite.java:57)
      

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            LuciferYang Yang Jie
            LuciferYang Yang Jie
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment