Details
-
Improvement
-
Status: Resolved
-
Minor
-
Resolution: Incomplete
-
2.2.0
-
None
Description
Following on from SPARK-20038: review SparkHadoopMapReduceWriter and ensure that it's failure handling code is itself resilient to follow on failures, especially in things like writer.close() and the abortTask/abortJob calls. That's to ensure as robust a cleanup as possible, and to stop the original exception getting lost.
At a quick glance
1. executeTask()'s catch logic should catch & log any failure in the writer.close()
2. The Hadoop commit protocol's abort* operation's can throw IOEs. Again, they need to be caught and logged
Should be testable with mocking, and worthwhile giving how important commit protocol resilience is.
Attachments
Issue Links
- is related to
-
SPARK-21549 Spark fails to complete job correctly in case of OutputFormat which do not write into hdfs
- Resolved