Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-13610

Conflict with Spark 3.1.2 scala version

Details

    • Bug
    • Status: Resolved
    • P2
    • Resolution: Duplicate
    • 2.34.0, 2.35.0
    • Not applicable
    • runner-spark
    • None

    Description

      When trying to use Beam with spark 3.1.2 we are running into this issue :

      InvalidClassException: scala.collection.mutable.WrappedArray 

      As explained here : https://www.mail-archive.com/issues@spark.apache.org/msg297820.html

      It's an incompatibility issue :  spark 3.1.2 is compiled with scala 2.12.10 but this issue is fixed only for scala >= 2.12.14.

       

      It seems that this issue can be fixed :

      => either by compiling spark & beam with a scala version >= 2.12.14

      => either by upgrading the spark version used by beam to 3.2.0, which can be compiled with scala 2.13

       

      Do you know if the switch to spark 3.2.0 is planned ?

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              syronanm Ronan SY
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: