Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33772 Build and Run Spark on Java 17
  3. SPARK-37073

Pass all UTs in `external/avro` with Java 17

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.0
    • 3.3.0
    • SQL
    • None

    Description

      Run `mvn clean install -pl external/avro` with Java 17

       

       

      Run completed in 43 seconds, 988 milliseconds.
      Total number of tests run: 283
      Suites: completed 14, aborted 0
      Tests: succeeded 281, failed 2, canceled 0, ignored 2, pending 0
      *** 2 TESTS FAILED ***
      

       

      - support user provided non-nullable avro schema for nullable catalyst schema without any null record *** FAILED ***
        "Job aborted due to stage failure: Task 1 in stage 144.0 failed 1 times, most recent failure: Lost task 1.0 in stage 144.0 (TID 250) (localhost executor driver): org.apache.spark.SparkException: Task failed while writing rows.
        	at org.apache.spark.sql.errors.QueryExecutionErrors$.taskFailedWhileWritingRowsError(QueryExecutionErrors.scala:516)
        	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:345)
        	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$20(FileFormatWriter.scala:252)
        	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
        	at org.apache.spark.scheduler.Task.run(Task.scala:136)
        	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:507)
        	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1468)
        	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:510)
        	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        	at java.base/java.lang.Thread.run(Thread.java:833)
        Caused by: org.apache.avro.file.DataFileWriter$AppendWriteException: java.lang.NullPointerException: Cannot invoke "Object.getClass()" because "datum" is null of string in string in field Name of test_schema in test_schema
        	at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:317)
        	at org.apache.spark.sql.avro.SparkAvroKeyRecordWriter.write(SparkAvroKeyOutputFormat.java:84)
        	at org.apache.spark.sql.avro.SparkAvroKeyRecordWriter.write(SparkAvroKeyOutputFormat.java:62)
        	at org.apache.spark.sql.avro.AvroOutputWriter.write(AvroOutputWriter.scala:84)
        	at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.write(FileFormatDataWriter.scala:175)
        	at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithMetrics(FileFormatDataWriter.scala:85)
        	at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithIterator(FileFormatDataWriter.scala:92)
        	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$1(FileFormatWriter.scala:328)
        	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1502)
        	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:335)
        	... 9 more
        Caused by: java.lang.NullPointerException: Cannot invoke "Object.getClass()" because "datum" is null of string in string in field Name of test_schema in test_schema
        	at org.apache.avro.generic.GenericDatumWriter.npe(GenericDatumWriter.java:184)
        	at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:160)
        	at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
        	at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:314)
        	... 18 more
        Caused by: java.lang.NullPointerException: Cannot invoke "Object.getClass()" because "datum" is null
        	at org.apache.avro.specific.SpecificDatumWriter.writeString(SpecificDatumWriter.java:68)
        	at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:151)
        	at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:83)
        	at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:158)
        	at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:221)
        	at org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:101)
        	at org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWriter.java:184)
        	at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:210)
        	at org.apache.avro.specific.SpecificDatumWriter.writeRecord(SpecificDatumWriter.java:83)
        	at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:131)
        	at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:83)
        	at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:158)
        	... 20 more
        
        Driver stacktrace:" did not contain "Caused by: java.lang.NullPointerException: null of string in string in field Name of test_schema in test_schema" (AvroSuite.scala:1074)
      

      Attachments

        Activity

          People

            LuciferYang Yang Jie
            LuciferYang Yang Jie
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: