Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26892

saveAsTextFile throws NullPointerException when null row present

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.4.0
    • 3.0.0
    • Spark Core
    • None

    Description

      We encoutered this problem in our production cluster, it can be reproduced by the following code:

      scala> sc.parallelize(Seq(1,null),1).saveAsTextFile("/tmp/foobar.dat")
      19/02/15 21:39:17 ERROR Utils: Aborting task
      java.lang.NullPointerException
      at org.apache.spark.rdd.RDD.$anonfun$saveAsTextFile$3(RDD.scala:1510)
      at scala.collection.Iterator$$anon$10.next(Iterator.scala:459)
      at org.apache.spark.internal.io.SparkHadoopWriter$.$anonfun$executeTask$1(SparkHadoopWriter.scala:129)
      at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1352)
      at org.apache.spark.internal.io.SparkHadoopWriter$.executeTask(SparkHadoopWriter.scala:127)
      at org.apache.spark.internal.io.SparkHadoopWriter$.$anonfun$write$1(SparkHadoopWriter.scala:83)
      at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
      at org.apache.spark.scheduler.Task.run(Task.scala:121)
      at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:425)
      at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1318)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:428)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      

      Attachments

        Issue Links

          Activity

            People

              liupengcheng liupengcheng
              liupengcheng liupengcheng
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: