Details
-
Bug
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
3.2.0
-
None
-
None
Description
I downloaded Spark 3.2 sourcecode from https://github.com/apache/spark/archive/refs/tags/v3.2.0.zip
and try building with the below commandÂ
./dev/make-distribution.sh --name without-hadoop --pip --r --tgz -Psparkr -Phive -Phive-thriftserver -Phadoop-provided -Pyarn -Dhadoop.version=3.1.4 -Pkubernetes
Then it gives the below errorÂ
[INFO] --- scala-maven-plugin:4.3.0:compile (scala-compile-first) @ spark-core_2.12 --- [INFO] Using incremental compilation using Mixed compile order [INFO] Compiler bridge file: /Users/JP28431/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.3.1-bin_2.12.15__52.0-1.3.1_20191012T045515.jar [INFO] compiler plugin: BasicArtifact(com.github.ghik,silencer-plugin_2.12.15,1.7.6,null) [INFO] Compiling 567 Scala sources and 104 Java sources to /Users/JP28431/Downloads/spark-3.2.0-github/core/target/scala-2.12/classes ... [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:38: object io is not a member of package org.apache.hadoop [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:2778: not found: type ArrayWritable [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:2777: not found: type Writable [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SSLOptions.scala:24: object conf is not a member of package org.apache.hadoop [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SSLOptions.scala:174: not found: type Configuration [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SecurityManager.scala:25: object io is not a member of package org.apache.hadoop [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SecurityManager.scala:26: object security is not a member of package org.apache.hadoop [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala:33: object fs is not a member of package org.apache.hadoop [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala:32: object conf is not a member of package org.apache.hadoop [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala:121: not found: type Configuration [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SecurityManager.scala:284: not found: value UserGroupInformation [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:41: object mapreduce is not a member of package org.apache.hadoop [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:40: object mapreduce is not a member of package org.apache.hadoop [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:39: object mapred is not a member of package org.apache.hadoop [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:37: object fs is not a member of package org.apache.hadoop [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:36: object conf is not a member of package org.apache.hadoop [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SecurityManager.scala:348: not found: type Credentials [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SecurityManager.scala:350: not found: value UserGroupInformation [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SecurityManager.scala:402: not found: type Text [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SerializableWritable.scala:22: object conf is not a member of package org.apache.hadoop [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SerializableWritable.scala:23: object io is not a member of package org.apache.hadoop [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SerializableWritable.scala:24: object io is not a member of package org.apache.hadoop [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SerializableWritable.scala:30: not found: type Writable [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SerializableWritable.scala:38: not found: type ObjectWritable [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SerializableWritable.scala:43: not found: type ObjectWritable [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala:20: object fs is not a member of package org.apache.hadoop [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala:21: object mapreduce is not a member of package org.apache.hadoop [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:210: not found: type Configuration [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:306: not found: type Configuration [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:26: object conf is not a member of package org.apache.hadoop [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:55: not found: type Configuration [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1188: not found: type InputFormat [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1161: not found: type InputFormat [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1120: not found: type InputFormat [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:925: not found: type TextInputFormat [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:925: not found: type LongWritable [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:925: not found: type Text [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:968: not found: value NewHadoopJob [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:971: not found: value NewFileInputFormat [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala:31: object mapreduce is not a member of package org.apache.hadoop [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/WholeTextFileRDD.scala:21: object io is not a member of package org.apache.hadoop [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/WholeTextFileRDD.scala:39: not found: type Text [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/WholeTextFileRDD.scala:39: not found: type Text [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/input/WholeTextFileInputFormat.scala:25: object mapreduce is not a member of package org.apache.hadoop [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/input/WholeTextFileInputFormat.scala:34: not found: type CombineFileInputFormat [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/input/WholeTextFileRecordReader.scala:21: object conf is not a member of package org.apache.hadoop [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/input/WholeTextFileRecordReader.scala:32: not found: type HConfigurable [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/WholeTextFileRDD.scala:35: not found: type Text [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/WholeTextFileRDD.scala:36: not found: type Text [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/WholeTextFileRDD.scala:20: object conf is not a member of package org.apache.hadoop [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/WholeTextFileRDD.scala:37: not found: type Configuration [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:976: not found: type Text [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:977: not found: type Text [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1020: not found: value NewHadoopJob [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1023: not found: value NewFileInputFormat [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/BinaryFileRDD.scala:22: object mapreduce is not a member of package org.apache.hadoop [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/input/PortableDataStream.scala:28: object mapreduce is not a member of package org.apache.hadoop [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/input/PortableDataStream.scala:39: not found: type CombineFileInputFormat [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/BinaryFileRDD.scala:20: object conf is not a member of package org.apache.hadoop [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/BinaryFileRDD.scala:34: not found: type Configuration [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1050: not found: type Configuration [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1237: not found: type NewInputFormat [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/input/FixedLengthBinaryInputFormat.scala:23: object mapreduce is not a member of package org.apache.hadoop [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/input/FixedLengthBinaryInputFormat.scala:43: not found: type FileInputFormat [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1210: not found: type NewInputFormat [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1053: not found: type LongWritable [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1053: not found: type BytesWritable [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1055: not found: type LongWritable [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1056: not found: type BytesWritable [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1087: not found: type JobConf [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1088: not found: type InputFormat [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1096: class FileSystem in package io cannot be accessed in package java.io [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala:34: object mapred is not a member of package org.apache.hadoop [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala:131: not found: type JobConf [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:29: object mapred is not a member of package org.apache.hadoop [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:112: not found: type JobConf [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:113: not found: type InputFormat [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:99: not found: type JobConf [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:100: not found: type InputFormat [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1128: not found: value FileSystem [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/util/SerializableConfiguration.scala:21: object conf is not a member of package org.apache.hadoop [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/util/SerializableConfiguration.scala:31: not found: type Configuration [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1132: not found: type JobConf [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1132: not found: value FileInputFormat [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1242: not found: type Configuration [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1247: not found: value FileSystem [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1251: not found: value NewHadoopJob [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1254: not found: value NewFileInputFormat [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala:73: not found: type InputFormat [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala:27: object conf is not a member of package org.apache.hadoop [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala:76: not found: type Configuration [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1278: not found: type Configuration [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1277: not found: type NewInputFormat [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1286: not found: value FileSystem [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1289: not found: type JobConf [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1315: not found: type SequenceFileInputFormat [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1375: not found: type SequenceFileInputFormat [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:3045: not found: type Writable [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1377: not found: type Writable [ERROR] [Error] /Users/JP28431/Downloads/spark-3.2.0-github/core/src/main/scala/org/apache/spark/SparkContext.scala:1378: not found: type Writable [INFO] [Info] : Any <: String? [INFO] [Info] : false [ERROR] 100 errors found [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary for Spark Project Parent POM 3.2.0: [INFO] [INFO] Spark Project Parent POM ........................... SUCCESS [ 4.964 s] [INFO] Spark Project Tags ................................. SUCCESS [ 8.381 s] [INFO] Spark Project Sketch ............................... SUCCESS [ 7.871 s] [INFO] Spark Project Local DB ............................. SUCCESS [ 2.191 s] [INFO] Spark Project Networking ........................... SUCCESS [ 4.879 s] [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 2.505 s] [INFO] Spark Project Unsafe ............................... SUCCESS [ 8.310 s] [INFO] Spark Project Launcher ............................. SUCCESS [ 1.990 s] [INFO] Spark Project Core ................................. FAILURE [ 45.343 s] [INFO] Spark Project ML Local Library ..................... SKIPPED [INFO] Spark Project GraphX ............................... SKIPPED [INFO] Spark Project Streaming ............................ SKIPPED [INFO] Spark Project Catalyst ............................. SKIPPED [INFO] Spark Project SQL .................................. SKIPPED [INFO] Spark Project ML Library ........................... SKIPPED [INFO] Spark Project Tools ................................ SKIPPED [INFO] Spark Project Hive ................................. SKIPPED [INFO] Spark Project REPL ................................. SKIPPED [INFO] Spark Project YARN Shuffle Service ................. SKIPPED [INFO] Spark Project YARN ................................. SKIPPED [INFO] Spark Project Kubernetes ........................... SKIPPED [INFO] Spark Project Hive Thrift Server ................... SKIPPED [INFO] Spark Project Assembly ............................. SKIPPED [INFO] Kafka 0.10+ Token Provider for Streaming ........... SKIPPED [INFO] Spark Integration for Kafka 0.10 ................... SKIPPED [INFO] Kafka 0.10+ Source for Structured Streaming ........ SKIPPED [INFO] Spark Project Examples ............................. SKIPPED [INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED [INFO] Spark Avro ......................................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 01:27 min [INFO] Finished at: 2022-01-24T11:07:30+09:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.3.0:compile (scala-compile-first) on project spark-core_2.12: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:4.3.0:compile failed.: CompileFailed -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <args> -rf :spark-core_2.12
is this a expected behavior or should be considered a bug ?
Attachments
Issue Links
- relates to
-
SPARK-35959 Add a new Maven profile "no-shaded-client" for older Hadoop 3.x versions
- Resolved