Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-18119

Handle sstable metadata stats file getting a new mtime after compaction has finished

    XMLWordPrintableJSON

Details

    • Degradation - Other Exception
    • Normal
    • Normal
    • Adhoc Test
    • All
    • None
    • Hide

      cci run, new unit tests

      Show
      cci run, new unit tests

    Description

      Due to a race between compaction finishing and compaction strategies getting reloaded there is a chance that we try to add both the new sstable and the old compacted sstable to the compaction strategy, and in the LCS case this can cause the old sstable to get sent to L0 to avoid overlap. This changes the mtime of the stats metadata file and if the node is shut down before the sstable is actually deleted from disk, we fail starting with the following exception:

      .../mockcf1-392b3ff07c5a11ed8c662f5760cb10b3/nb_txn_compaction_3983c030-7c5a-11ed-8c66-2f5760cb10b3.log
      REMOVE:[.../data/TransactionLogsTest/mockcf1-392b3ff07c5a11ed8c662f5760cb10b3/nb-0-big-,1671096247000,5][4003386800]
               ***Unexpected files detected for sstable [nb-0-big-]: last update time [Thu Dec 15 10:24:09 CET 2022] (1671096249000) should have been [Thu Dec 15 10:24:07 CET 2022] (1671096247000)
      ADD:[.../data/TransactionLogsTest/mockcf1-392b3ff07c5a11ed8c662f5760cb10b3/nb-2-big-,0,5][319189529]
      

      A workaround for this (until we properly fix the way compaction strategies get notified about sstable changes) is to ignore the timestamp of the STATS component when cleaning up compaction leftovers on startup.

      Attachments

        Activity

          People

            marcuse Marcus Eriksson
            marcuse Marcus Eriksson
            Josh McKenzie
            Jon Meredith, Josh McKenzie
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 40m
                40m