Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-14239

OutOfMemoryError when bootstrapping with less than 100GB RAM

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Normal
    • Resolution: Unresolved
    • None
    • None
    • Normal

    Description

      Hi, I face an issue when bootstrapping a Node having less than 100GB RAM on our 10 Node C* 3.11.1 Cluster.

      During bootstrap, when I watch the cassandra.log I observe a growth in JVM Heap Old Gen which gets not significantly freed up any more.

      I know that JVM collects on Old Gen only when really needed. I can see collections, but there is always a remainder which seems to grow forever without ever getting freed.

      After the Node successfully Joined the Cluster, I can remove the extra RAM I have given it for bootstrapping without any further effect.

      It feels like Cassandra will not forget about every single byte streamed over the Network over time during bootstrapping, - which would be a memory leak and a major problem, too.

      I was able to produce a HeapDumpOnOutOfMemoryError from a 56GB Node (40 GB assigned JVM Heap). YourKit Profiler shows huge amount of Memory allocated for org.apache.cassandra.db.Memtable (22 GB) org.apache.cassandra.db.rows.BufferCell (19 GB) and java.nio.HeapByteBuffer (11 GB)

      Attachments

        1. Objects-by-class.csv
          274 kB
          Jürgen Albersdorfer
        2. stack-traces.txt
          196 kB
          Jürgen Albersdorfer
        3. Objects-with-biggest-retained-size.csv
          3 kB
          Jürgen Albersdorfer
        4. jvm.options
          9 kB
          Jürgen Albersdorfer
        5. cassandra-env.sh
          12 kB
          Jürgen Albersdorfer
        6. cassandra.yaml
          56 kB
          Jürgen Albersdorfer
        7. jvm_opts.txt
          5 kB
          Jürgen Albersdorfer
        8. gc.log.0.current.zip
          620 kB
          Jürgen Albersdorfer
        9. gc.log.201804111141.zip
          274 kB
          Jürgen Albersdorfer
        10. gc.log.0.201804111524.zip
          152 kB
          Jürgen Albersdorfer
        11. Selection_420.png
          94 kB
          Sergey Kirillov
        12. Selection_421.png
          41 kB
          Sergey Kirillov
        13. dstat.png
          94 kB
          Sergey Kirillov

        Issue Links

          Activity

            People

              Unassigned Unassigned
              jalbersdorfer Jürgen Albersdorfer
              Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated: