Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3603

InvalidClassException on a Linux VM - probably problem with serialization

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Cannot Reproduce
    • 1.0.0, 1.1.0
    • None
    • Deploy

    Description

      I have a Scala app connecting to a standalone Spark cluster. It works fine on Windows or on a Linux VM; however, when I try to run the app and the Spark cluster on another Linux VM (the same Linux kernel, Java and Spark - tested for versions 1.0.0 and 1.1.0) I get the below exception. This looks kind of similar to the Big-Endian (IBM Power7) Spark Serialization issue (SPARK-2018), but... my system is definitely little endian and I understand the big endian issue should be already fixed in Spark 1.1.0 anyway. I'd appreaciate your help.

      01:34:53.251 WARN [Result resolver thread-0][TaskSetManager] Lost TID 2 (task 1.0:2)
      01:34:53.278 WARN [Result resolver thread-0][TaskSetManager] Loss was due to java.io.InvalidClassException
      java.io.InvalidClassException: scala.reflect.ClassTag$$anon$1; local class incompatible: stream classdesc serialVersionUID = -4937928798201944954, local class serialVersionUID = -8102093212602380348
      at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
      at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620)
      at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1515)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1769)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
      at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1913)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
      at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1913)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
      at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1891)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
      at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1913)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
      at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1913)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
      at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1891)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
      at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1913)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
      at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1913)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
      at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1704)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1342)
      at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1913)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
      at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1913)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
      at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1913)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
      at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1891)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
      at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1913)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
      at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:63)
      at org.apache.spark.scheduler.ShuffleMapTask$.deserializeInfo(ShuffleMapTask.scala:63)
      at org.apache.spark.scheduler.ShuffleMapTask.readExternal(ShuffleMapTask.scala:135)
      at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1835)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1794)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
      at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:63)
      at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:85)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:169)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:724)

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              tomasz Tomasz Dudziak
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: