Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-18176

Merged SSTable files not reclaimed by OS

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Normal
    • Resolution: Fixed
    • 4.0.9
    • Local/Compaction
    • None
    • Degradation - Resource Management
    • Normal
    • Normal
    • User Report
    • All
    • None

    Description

      (EDIT: Looks like this is masked by the lack of backport of CASSANDRA-17205 just on the 4.0 line - will backport that and block here on it)

      After upgrading to Cassandra 4.x (4.0.1 and 4.0.5) we've noticed that at times after a compaction deleted sstables diskspace doesn't get reclaimed by the OS until the cassandra process is restarted (which kinda points at some sort of resource leak), I do not recall this happening in cassandra 3, at least not to such degree.

      We've seen the behavior in multiple clusters with different schemas, access patterns and consistency levels at somewhat "random" points in time, the only interesting thing is that there were active repair sessions at the time affecting the node, keyspace and table.

      $ date +%Y-%m-%d
2023-01-17

$ nodetool version
ReleaseVersion: 4.0.5

      $ lsof +L1 | grep cassandra | grep myawesomecluster | wc -l
2772

$ lsof +L1 | grep cassandra | grep myawesomecluster | tail -n1
java      59003 cassandra *979u   REG  253,8        10     0 1208053768 /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274426-big-Digest.crc32 (deleted)

      $ grep 2274426 /cassandra/systemlog.log
INFO  [Stream-Deserializer-/10.214.79.62:randomport-b904af67] 2023-01-15 13:06:24,969 BigTableZeroCopyWriter.java:203 - Writing component DATA to /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274426-big-Data.db length 2.900KiB
INFO  [Stream-Deserializer-/10.214.79.62:randomport-b904af67] 2023-01-15 13:06:24,969 BigTableZeroCopyWriter.java:213 - Block Writing component to /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274426-big-Data.db length 2.900KiB
INFO  [Stream-Deserializer-/10.214.79.62:randomport-b904af67] 2023-01-15 13:06:24,970 BigTableZeroCopyWriter.java:203 - Writing component PRIMARY_INDEX to /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274426-big-Index.db length 3.739KiB
INFO  [Stream-Deserializer-/10.214.79.62:randomport-b904af67] 2023-01-15 13:06:24,970 BigTableZeroCopyWriter.java:213 - Block Writing component to /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274426-big-Index.db length 3.739KiB
INFO  [Stream-Deserializer-/10.214.79.62:randomport-b904af67] 2023-01-15 13:06:24,970 BigTableZeroCopyWriter.java:203 - Writing component STATS to /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274426-big-Statistics.db length 5.062KiB
INFO  [Stream-Deserializer-/10.214.79.62:randomport-b904af67] 2023-01-15 13:06:24,970 BigTableZeroCopyWriter.java:213 - Block Writing component to /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274426-big-Statistics.db length 5.062KiB
INFO  [Stream-Deserializer-/10.214.79.62:randomport-b904af67] 2023-01-15 13:06:24,971 BigTableZeroCopyWriter.java:203 - Writing component COMPRESSION_INFO to /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274426-big-CompressionInfo.db length 0.054KiB
INFO  [Stream-Deserializer-/10.214.79.62:randomport-b904af67] 2023-01-15 13:06:24,971 BigTableZeroCopyWriter.java:213 - Block Writing component to /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274426-big-CompressionInfo.db length 0.054KiB
INFO  [Stream-Deserializer-/10.214.79.62:randomport-b904af67] 2023-01-15 13:06:24,971 BigTableZeroCopyWriter.java:203 - Writing component FILTER to /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274426-big-Filter.db length 0.031KiB
INFO  [Stream-Deserializer-/10.214.79.62:randomport-b904af67] 2023-01-15 13:06:24,971 BigTableZeroCopyWriter.java:213 - Block Writing component to /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274426-big-Filter.db length 0.031KiB
INFO  [Stream-Deserializer-/10.214.79.62:randomport-b904af67] 2023-01-15 13:06:24,972 BigTableZeroCopyWriter.java:203 - Writing component SUMMARY to /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274426-big-Summary.db length 0.436KiB
INFO  [Stream-Deserializer-/10.214.79.62:randomport-b904af67] 2023-01-15 13:06:24,972 BigTableZeroCopyWriter.java:213 - Block Writing component to /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274426-big-Summary.db length 0.436KiB
INFO  [Stream-Deserializer-/10.214.79.62:randomport-b904af67] 2023-01-15 13:06:24,972 BigTableZeroCopyWriter.java:203 - Writing component DIGEST to /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274426-big-Digest.crc32 length 0.010KiB
INFO  [Stream-Deserializer-/10.214.79.62:randomport-b904af67] 2023-01-15 13:06:24,972 BigTableZeroCopyWriter.java:213 - Block Writing component to /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274426-big-Digest.crc32 length 0.010KiB
INFO  [Stream-Deserializer-/10.214.79.62:randomport-b904af67] 2023-01-15 13:06:24,974 SSTableReaderBuilder.java:351 - Opening /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274426-big (2.900KiB)
INFO  [CompactionExecutor:54141] 2023-01-15 13:06:24,978 CompactionTask.java:150 - Compacting (6a3e8320-94d5-11ed-b2c8-7b967b642f39) [/cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274426-big-Data.db:level=0, /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274422-big-Data.db:level=0, /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274411-big-Data.db:level=0, /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274423-big-Data.db:level=0, /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274410-big-Data.db:level=0, /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274412-big-Data.db:level=0, /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274413-big-Data.db:level=0, /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274414-big-Data.db:level=0, /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274419-big-Data.db:level=0, /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274425-big-Data.db:level=0, /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274424-big-Data.db:level=0, /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274418-big-Data.db:level=0, /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274421-big-Data.db:level=0, /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274420-big-Data.db:level=0, /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274416-big-Data.db:level=0, /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274417-big-Data.db:level=0, /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274415-big-Data.db:level=0, ]
INFO  [NonPeriodicTasks:1] 2023-01-15 13:06:25,254 SSTable.java:111 - Deleting sstable: /cassandra-data/data/myawesomecluster/schema_one-randomchecksum/nb-2274426-big

      $ nodetool compactionhistory | grep '2023-01-15T13:06' 
75032a40-94d5-11ed-b2c8-7b967b642f39 myawesomecluster    schema_one          2023-01-15T13:06:43.044 53958242   52853746   {1:478098, 2:11313, 3:1765, 4:287, 5:185, 6:88, 7:39, 8:22, 9:13, 10:7, 11:3, 14:1}
6a4539e0-94d5-11ed-b2c8-7b967b642f39 myawesomecluster    schema_one          2023-01-15T13:06:25.022 107577     30110      {1:25, 2:66, 3:124, 4:16, 5:66, 6:15, 9:1}
6a1b43b0-94d5-11ed-b2c8-7b967b642f39 myawesomecluster    schema_one          2023-01-15T13:06:24.747 91018      29868      {1:16, 2:127, 3:148, 4:5, 5:7, 6:6, 7:3, 8:1, 10:1}
6a063510-94d5-11ed-b2c8-7b967b642f39 myawesomecluster    schema_one          2023-01-15T13:06:24.609 767        366        {2:2}
6a0523a0-94d5-11ed-b2c8-7b967b642f39 myawesomecluster    schema_one          2023-01-15T13:06:24.602 87020      27917      {1:73, 2:38, 3:96, 4:15, 5:67, 6:1}
69c5a9a0-94d5-11ed-b2c8-7b967b642f39 myawesomecluster    schema_one          2023-01-15T13:06:24.186 45345      25814      {1:117, 2:150, 3:3}
6956bb30-94d5-11ed-b2c8-7b967b642f39 myawesomecluster    schema_two          2023-01-15T13:06:23.459 8925831    8904158    {1:102662, 2:120}
5d886c40-94d5-11ed-b2c8-7b967b642f39 myawesomecluster    schema_two          2023-01-15T13:06:03.652 8901698    8901269    {1:102606, 2:9}
5d02c180-94d5-11ed-b2c8-7b967b642f39 myawesomecluster    schema_two          2023-01-15T13:06:02.776 8976310    8900384    {1:102844, 2:324}
5bf9dd00-94d5-11ed-b2c8-7b967b642f39 system_distributed repair_history        2023-01-15T13:06:01.040 4848124    4847506    {4:2}
5bc27950-94d5-11ed-b2c8-7b967b642f39 system_distributed parent_repair_history 2023-01-15T13:06:00.677 351877     351135     {1:2383, 2:1}

      Attachments

        1. cassandra_18332-20230415.txt
          23 kB
          Pere Balaguer

        Issue Links

          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. CASSANDRA-18256 Backport CASSANDRA-17205 (remove strong self-ref in tidier) to all supported lines

          • Normal -
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. CASSANDRA-18332 Backport CASSANDRA-17205 to 4.0 branch (strong ref leak)

          • Normal -
          • Resolved

          Activity

            People

              Unassigned Unassigned
              pbalaguer Pere Balaguer
              Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: