Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33772 Build and Run Spark on Java 17
  3. SPARK-36900

"SPARK-36464: size returns correct positive number even with over 2GB data" will oom with JDK17

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.3.0
    • 3.2.1, 3.3.0
    • Spark Core
    • None

    Description

      Execute

       

      build/mvn clean install  -pl core -am -Dtest=none -DwildcardSuites=org.apache.spark.util.io.ChunkedByteBufferOutputStreamSuite
      

      with JDK 17,

      ChunkedByteBufferOutputStreamSuite:
      - empty output
      - write a single byte
      - write a single near boundary
      - write a single at boundary
      - single chunk output
      - single chunk output at boundary size
      - multiple chunk output
      - multiple chunk output at boundary size
      *** RUN ABORTED ***
        java.lang.OutOfMemoryError: Java heap space
        at java.base/java.lang.Integer.valueOf(Integer.java:1081)
        at scala.runtime.BoxesRunTime.boxToInteger(BoxesRunTime.java:67)
        at org.apache.spark.util.io.ChunkedByteBufferOutputStream.allocateNewChunkIfNeeded(ChunkedByteBufferOutputStream.scala:87)
        at org.apache.spark.util.io.ChunkedByteBufferOutputStream.write(ChunkedByteBufferOutputStream.scala:75)
        at java.base/java.io.OutputStream.write(OutputStream.java:127)
        at org.apache.spark.util.io.ChunkedByteBufferOutputStreamSuite.$anonfun$new$22(ChunkedByteBufferOutputStreamSuite.scala:127)
        at org.apache.spark.util.io.ChunkedByteBufferOutputStreamSuite$$Lambda$179/0x00000008011a75d8.apply(Unknown Source)
        at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
        at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
        at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
      

       

       

      Attachments

        Activity

          People

            LuciferYang Yang Jie
            LuciferYang Yang Jie
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: