Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25737

Remove JavaSparkContextVarargsWorkaround and standardize union() methods

    XMLWordPrintableJSON

Details

    • Task
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • Spark Core
    • Hide
      For release notes:

      Before Spark 3.0, JavaSparkContext exposed union() methods with signatures like union(JavaRDD, List<JavaRDD>). This was a workaround for a compiler issue that is since resolved. The signature of union() methods in JavaSparkContext has changed to simple varargs: union(JavaRDD*).
      Show
      For release notes: Before Spark 3.0, JavaSparkContext exposed union() methods with signatures like union(JavaRDD, List<JavaRDD>). This was a workaround for a compiler issue that is since resolved. The signature of union() methods in JavaSparkContext has changed to simple varargs: union(JavaRDD*).

    Description

      In ancient times in 2013, JavaSparkContext got a superclass JavaSparkContextVarargsWorkaround to deal with some Scala 2.7 issue: http://www.scala-archive.org/Workaround-for-implementing-java-varargs-in-2-7-2-final-td1944767.html#a1944772

      I believe this was really resolved by the @varags annotation in Scala 2.9. 

      I believe we can now remove this workaround. union(JavaRDD, List<JavaRDD>) should just be union(JavaRDD*) and likewise for JavaPairRDD, JavaDoubleRDD.

       

      Attachments

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. SPARK-31788 Error when creating UnionRDD of PairRDDs

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          links to

          Pull request #22729 [Github] Pull Request #22729 (srowen)

          Activity

            People

              srowen Sean R. Owen
              srowen Sean R. Owen
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: