Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-10449

OOM on bootstrap after long GC pause

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Normal
    • Resolution: Duplicate
    • 2.1.x
    • None
    • Ubuntu 14.04, AWS

    • Normal

    Description

      I have a 20-node cluster (i2.4xlarge) with vnodes (default of 256) and 500-700GB per node. SSTable counts are <10 per table. I am attempting to provision additional nodes, but bootstrapping OOMs every time after about 10 hours with a sudden long GC pause:

      INFO  [Service Thread] 2015-10-05 23:33:33,373 GCInspector.java:252 - G1 Old Generation GC in 1586126ms.  G1 Old Gen: 49213756976 -> 49072277176;
      ...
      ERROR [MemtableFlushWriter:454] 2015-10-05 23:33:33,380 CassandraDaemon.java:223 - Exception in thread Thread[MemtableFlushWriter:454,5,main]
      java.lang.OutOfMemoryError: Java heap space
      

      I have tried increasing max heap to 48G just to get through the bootstrap, to no avail.

      Attachments

        1. GCpath.txt
          13 kB
          Mikhail Stepura
        2. threads.txt
          83 kB
          Mikhail Stepura
        3. heap_dump.png
          103 kB
          Robert Strickland
        4. thread_dump.log
          221 kB
          Robert Strickland
        5. system.log.10-05
          18.89 MB
          Robert Strickland

        Issue Links

          Activity

            People

              Unassigned Unassigned
              rstrickland Robert Strickland
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: