Details
-
Bug
-
Status: Resolved
-
Normal
-
Resolution: Fixed
-
None
-
None
-
None
-
Normal
Description
If there is an error opening a commit log segment, the thread dies, but it doesn't stop the whole node.
Got the following on a node:
ERROR [COMMIT-LOG-ALLOCATOR] 2015-05-29 17:06:45,722 CommitLog.java:397 - Failed managing commit log segments. Commit disk failure policy is stop; terminating thread org.apache.cassandra.io.FSWriteError: java.io.FileNotFoundException: /var/lib/cassandra/commitlog/CommitLog-4-1432937194590.log (Permission denied) at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:177) ~[cassandra-all-2.1.5.jar:2.1.5] at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$4.call(CommitLogSegmentManager.java:397) ~[cassandra-all-2.1.5.jar:2.1.5] at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$4.call(CommitLogSegmentManager.java:394) ~[cassandra-all-2.1.5.jar:2.1.5] at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$1.runMayThrow(CommitLogSegmentManager.java:152) ~[cassandra-all-2.1.5.jar:2.1.5] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) [cassandra-all-2.1.5.jar:2.1.5] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45] Caused by: java.io.FileNotFoundException: /var/lib/cassandra/commitlog/CommitLog-4-1432937194590.log (Permission denied) at java.io.RandomAccessFile.open0(Native Method) ~[na:1.8.0_45] at java.io.RandomAccessFile.open(RandomAccessFile.java:316) ~[na:1.8.0_45] at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243) ~[na:1.8.0_45] at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:155) ~[cassandra-all-2.1.5.jar:2.1.5] ... 5 common frames omitted
And the node stayed kind of up, didn't notice something wrong until the node died OOM because some threads were dead and others weren't.
Attachments
Issue Links
- duplicates
-
CASSANDRA-8515 Commit log stop policy not enforced correctly during startup
- Resolved