Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-2507

Compile error of streaming project with 2.0.0-cdh4.6.0

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 0.9.0, 0.9.1, 1.0.0, 1.0.1, 1.0.2
    • None
    • DStreams
    • None
    • RedHat 5.3
      2.0.0-cdh4.6.0
      enable yarn
      java version "1.6.0_45"

    Description

      Hi,

      When compiling with

      ./make-distribution.sh --hadoop 2.0.0-cdh4.6.0 --with-yarn --tgz

      I have the following errors on streaming java api:

      Version is 0.9.0-incubating
      Making spark-0.9.0-incubating-hadoop_2.0.0-cdh4.6.0-bin.tar.gz
      Hadoop version set to 2.0.0-cdh4.6.0
      YARN enabled
      [info] Loading project definition from /root/spark-source/project/project
      [info] Loading project definition from /root/spark-source/project
      [info] Set current project to root (in build file:/root/spark-source/)
      [info] Compiling 1 Scala source to /root/spark-source/streaming/target/scala-2.10/classes...
      [error] /root/spark-source/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:57: type mismatch;
      [error] found : org.apache.spark.streaming.dstream.DStream[(K, V)]
      [error] required: org.apache.spark.streaming.api.java.JavaPairDStream[K,V]
      [error] Note: implicit method fromPairDStream is not applicable here because it comes after the application point and it lacks an explicit result type
      [error] dstream.filter((x => f.booleanValue()))
      [error] ^
      [error] /root/spark-source/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:60: type mismatch;
      [error] found : org.apache.spark.streaming.dstream.DStream[(K, V)]
      [error] required: org.apache.spark.streaming.api.java.JavaPairDStream[K,V]
      [error] Note: implicit method fromPairDStream is not applicable here because it comes after the application point and it lacks an explicit result type
      [error] def cache(): JavaPairDStream[K, V] = dstream.cache()
      [error] ^
      [error] /root/spark-source/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:63: type mismatch;
      [error] found : org.apache.spark.streaming.dstream.DStream[(K, V)]
      [error] required: org.apache.spark.streaming.api.java.JavaPairDStream[K,V]
      [error] Note: implicit method fromPairDStream is not applicable here because it comes after the application point and it lacks an explicit result type
      [error] def persist(): JavaPairDStream[K, V] = dstream.persist()
      [error] ^

      ......

      [error] /root/spark-source/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:669: type mismatch;
      [error] found : org.apache.spark.streaming.dstream.DStream[(K, (com.google.common.base.Optional[V], W))]
      [error] required: org.apache.spark.streaming.api.java.JavaPairDStream[K,(com.google.common.base.Optional[V], W)]
      [error] Note: implicit method fromPairDStream is not applicable here because it comes after the application point and it lacks an explicit result type
      [error] joinResult.mapValues

      Unknown macro: {case (v, w) => (JavaUtils.optionToOptional(v), w)}

      [error] ^
      [error] 44 errors found
      [error] (streaming/compile:compile) Compilation failed

      Here is a simple PR fix this problem: https://github.com/apache/spark/pull/153

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              gzm55 James Z.M. Gao
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: