Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Duplicate
-
2.1.0
-
None
-
None
Description
(this has been reported on the user list in January and previously by a different user)
------------------------------------------------------------------------------------
My code works fine with Storm 1.x but the workers crash constantly with Storm 2.x
Some exceptions look like
com.esotericsoftware.kryo.KryoException: Buffer underflow.
at com.esotericsoftware.kryo.io.Input.require(Input.java:199) ~[kryo-3.0.3.jar:?]
at com.esotericsoftware.kryo.io.Input.readUtf8_slow(Input.java:575) ~[kryo-3.0.3.jar:?]
at com.esotericsoftware.kryo.io.Input.readUtf8(Input.java:553) ~[kryo-3.0.3.jar:?]
at com.esotericsoftware.kryo.io.Input.readString(Input.java:483) ~[kryo-3.0.3.jar:?]
whereas others are
2020-01-21 11:13:39.368 o.a.s.m.n.StormServerHandler Netty-server-localhost-6701-worker-1 [ERROR] server errors in handling the request
{{_com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 95
at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:137) ~[kryo-3.0.3.jar:?]
at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670) ~[kryo-3.0.3.jar:?]
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:781) ~[kryo-3.0.3.jar:?]
at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:153) ~[kryo-3.0.3.jar:?]
at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39) ~[kryo-3.0.3.jar:?]
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793) ~[kryo-3.0.3.jar:?]
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134) ~[kryo-3.0.3.jar:?]
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40) ~[kryo-3.0.3.jar:?]
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:689) ~[kryo-3.0.3.jar:?]
at org.apache.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:31) ~[storm-client-2.1.0.jar:2.1.0]_}}
the class ID values are random integers.
{{}}
The tuples in my topologies contain mostly standard classes like String; the only exception is the following class
https://github.com/DigitalPebble/storm-crawler/blob/2.x/core/src/main/java/com/digitalpebble/stormcrawler/Metadata.java#L41
for which we specify a custom serialization for Kryo.
My configurations contain
topology.kryo.register: |
|
to register the custom class.
The user who had reported the problem first also used custom serialization.