Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Not A Problem
-
3.0.0
-
None
-
None
Description
Unless we configure the property,
spark.local.dir /tmp
following error occurs:
*20/01/21 08:33:17 INFO SparkEnv: Registering BlockManagerMasterHeartbeat* *20/01/21 08:33:17 ERROR DiskBlockManager: Failed to create local dir in /var/data/spark-284c6844-8969-4288-9a6b-b72679c5b8e4. Ignoring this directory.* *java.io.IOException: Failed to create a temp directory (under /var/data/spark-284c6844-8969-4288-9a6b-b72679c5b8e4) after 10 attempts!* *at org.apache.spark.util.Utils$.createDirectory(Utils.scala:304)* *at org.apache.spark.storage.DiskBlockManager.$anonfun$createLocalDirs$1(DiskBlockManager.scala:164)* *at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)* *at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)*
I have not yet fully understood the root cause, will post my findings once it is clear.