Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-19158

ml.R example fails in yarn-cluster mode due to lacks of e1071 package

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.1.1, 2.2.0
    • Examples
    • None

    Description

      ml.R application fails in spark2 with yarn-cluster mode.

      spark-submit --master yarn-cluster examples/src/main/r/ml/ml.R 
      application log
      17/01/03 04:35:30 INFO MemoryStore: Block broadcast_88 stored as values in memory (estimated size 6.8 KB, free 407.6 MB)
      17/01/03 04:35:30 INFO BufferedStreamThread: Error : requireNamespace("e1071", quietly = TRUE) is not TRUE
      17/01/03 04:35:30 ERROR Executor: Exception in task 0.0 in stage 65.0 (TID 65)
      org.apache.spark.SparkException: R computation failed with
       Error : requireNamespace("e1071", quietly = TRUE) is not TRUE
      	at org.apache.spark.api.r.RRunner.compute(RRunner.scala:108)
      	at org.apache.spark.api.r.BaseRRDD.compute(RRDD.scala:50)
      	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
      	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
      	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
      	at org.apache.spark.scheduler.Task.run(Task.scala:99)
      	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      17/01/03 04:35:30 INFO CoarseGrainedExecutorBackend: Got assigned task 68
      17/01/03 04:35:30 INFO Executor: Running task 3.0 in stage 65.0 (TID 68)
      17/01/03 04:35:30 INFO BufferedStreamThread: Error : requireNamespace("e1071", quietly = TRUE) is not TRUE
      17/01/03 04:35:30 ERROR Executor: Exception in task 3.0 in stage 65.0 (TID 68)
      org.apache.spark.SparkException: R computation failed with
       Error : requireNamespace("e1071", quietly = TRUE) is not TRUE
      Error : requireNamespace("e1071", quietly = TRUE) is not TRUE
      	at org.apache.spark.api.r.RRunner.compute(RRunner.scala:108)
      	at org.apache.spark.api.r.BaseRRDD.compute(RRDD.scala:50)
      	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
      	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
      	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
      	at org.apache.spark.scheduler.Task.run(Task.scala:99)
      	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      17/01/03 04:35:30 INFO CoarseGrainedExecutorBackend: Got assigned task 70
      

      Attachments

        Activity

          People

            yanboliang Yanbo Liang
            yeshavora Yesha Vora
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: