Minor Description
Testing some failure scenarios, and JdbcUtils throws an IllegalStateException instead of the expected SQLException:
scala> org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils.saveTable(prodtbl,url3,"DB2.D_ITEM_INFO",prop1) 17/04/03 17:19:35 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 1) java.lang.IllegalStateException: Cause already initialized .at java.lang.Throwable.setCause(Throwable.java:365) .at java.lang.Throwable.initCause(Throwable.java:341) .at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.savePartition(JdbcUtils.scala:241) .at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$saveTable$1.apply(JdbcUtils.scala:300) .at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$saveTable$1.apply(JdbcUtils.scala:299) .at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:902) .at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:902) .at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1899) .at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1899) .at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70) .at org.apache.spark.scheduler.Task.run(Task.scala:86) .at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) .at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153 .at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628 .at java.lang.Thread.run(Thread.java:785)
The code in JdbcUtils.savePartition has
} catch { case e: SQLException => val cause = e.getNextException if (cause != null && e.getCause != cause) { if (e.getCause == null) { e.initCause(cause) } else { e.addSuppressed(cause) } }
According to Throwable Java doc, initCause() throws an IllegalStateException "if this throwable was created with Throwable(Throwable) or Throwable(String,Throwable), or this method has already been called on this throwable". The code does check whether cause is null before initializing it. However, getCause() "returns the cause of this throwable or null if the cause is nonexistent or unknown." In other words, null is returned if cause already exists (which would result in IllegalStateException) but is unknown.
