Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-47660

Memory leak in StructType column's .dropFields method

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.5.1
    • None
    • Java API, SQL
    • None

    Description

      We noticed that when we're using the .dropFields method, for example by calling

      df.withColum(column, col(column).dropFields("fieldToDrop")) the heap "explodes". we noticed it when our integration tests, for which 512MB heap is always enough started crashing with OutOfMemoryException (even when we increased the heap size to 8GB) and after a lot of trial and error we found that when we remove the .dropFields method the heap goes back down 

      Attachments

        Activity

          People

            Unassigned Unassigned
            eylonronen Eylon Ronen
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: