[CASSANDRA-14239] OutOfMemoryError when bootstrapping with less than 100GB RAM - ASF JIRA

Log work

Agile Board

Rank to Top

Rank to Bottom

Attach files

Attach Screenshot

Bulk Copy Attachments

Bulk Move Attachments

Add vote

Voters

Watch issue

Watchers

Create sub-task

Convert to sub-task

Move

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Open
Priority: Normal
Resolution: Unresolved
Fix Version/s: None
Component/s: Feature/Materialized Views
Labels:
None
Environment:
Hide

Details of the bootstrapping Node

ProLiant BL460c G7

56GB RAM

2x 146GB 10K HDD (One dedicated for Commitlog, one for Data, Hints and saved_caches)

CentOS 7.4 on SD-Card

/tmp and /var/log on tmpfs

Oracle JDK 1.8.0_151

Cassandra 3.11.1

Cluster

10 existing Nodes (Up and Normal)
Show
Details of the bootstrapping Node ProLiant BL460c G7 56GB RAM 2x 146GB 10K HDD (One dedicated for Commitlog, one for Data, Hints and saved_caches) CentOS 7.4 on SD-Card /tmp and /var/log on tmpfs Oracle JDK 1.8.0_151 Cassandra 3.11.1 Cluster 10 existing Nodes (Up and Normal)

Severity:
Normal
Since Version:

3.11.1

Description

Hi, I face an issue when bootstrapping a Node having less than 100GB RAM on our 10 Node C* 3.11.1 Cluster.

During bootstrap, when I watch the cassandra.log I observe a growth in JVM Heap Old Gen which gets not significantly freed up any more.

I know that JVM collects on Old Gen only when really needed. I can see collections, but there is always a remainder which seems to grow forever without ever getting freed.

After the Node successfully Joined the Cluster, I can remove the extra RAM I have given it for bootstrapping without any further effect.

It feels like Cassandra will not forget about every single byte streamed over the Network over time during bootstrapping, - which would be a memory leak and a major problem, too.

I was able to produce a HeapDumpOnOutOfMemoryError from a 56GB Node (40 GB assigned JVM Heap). YourKit Profiler shows huge amount of Memory allocated for org.apache.cassandra.db.Memtable (22 GB) org.apache.cassandra.db.rows.BufferCell (19 GB) and java.nio.HeapByteBuffer (11 GB)