Details
-
Bug
-
Status: Closed
-
Minor
-
Resolution: Fixed
-
1.1.0, 1.1.1
-
None
Description
1. when throws MetadataFetchFailedException, it will create a FetchFailedException, which BlockManagerId is null.
FetchFailedException(null, shuffleId, -1, reduceId)
2. JsonProtocol.TaskEndReasonToJson, mark BlockManagerId as JNothing
3. while use JsonProtocol.TaskEndReasonFromJson, it didn't take JNothing into account, so BlockManagerIdFromJson will throw exception
error log belows:
org.json4s.package$MappingException: Did not find value which can be converted into java.lang.String
at org.json4s.reflect.package$.fail(package.scala:96)
at org.json4s.Extraction$.convert(Extraction.scala:554)
at org.json4s.Extraction$.extract(Extraction.scala:331)
at org.json4s.Extraction$.extract(Extraction.scala:42)
at org.json4s.ExtractableJsonAstNode.extract(ExtractableJsonAstNode.scala:21)
at org.apache.spark.util.JsonProtocol$.blockManagerIdFromJson(JsonProtocol.scala:662)
at org.apache.spark.util.JsonProtocol$.taskEndReasonFromJson(JsonProtocol.scala:643)
at org.apache.spark.util.JsonProtocol$.taskEndFromJson(JsonProtocol.scala:450)
at org.apache.spark.util.JsonProtocol$.sparkEventFromJson(JsonProtocol.scala:410)
at org.apache.spark.scheduler.ReplayListenerBus$$anonfun$replay$2$$anonfun$apply$1.apply(ReplayListenerBus.scala:71)
at org.apache.spark.scheduler.ReplayListenerBus$$anonfun$replay$2$$anonfun$apply$1.apply(ReplayListenerBus.scala:69)