Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-9914

Millions of fake pending compaction tasks + high CPU

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Normal
    • Resolution: Duplicate
    • None
    • None
    • None
    • CentOS

    • Normal

    Description

      We have a 3-node test cluster (initially running 2.1.8) with zero traffic and about 10GB of data on each node. It's showing millions of pending compaction tasks (but no actual work in progress), and the CPUs are pegged on all three nodes. The task count goes down rapidly, but then jumps back up again seconds later. All tables are set to STCS. The issue persists after restart, but takes a few minutes before it becomes a problem. SSTable counts are below 10 for every table. We're also seeing 20s Old Gen GC pauses about every 2-3 mins.

      This started happening after bulk loading some old data. We started seeing very long GC pauses (sometimes 30 min or more) that would bring down the nodes. We then truncated this table, which resulted in the current behavior. We attempted to roll back our cluster to 2.1.7 patched with CASSANDRA-9637, but we observed the same behavior.

      Attachments

        1. high_pending_compactions.txt
          3 kB
          Robert Strickland
        2. cass_high_cpu.png
          212 kB
          Robert Strickland

        Issue Links

          Activity

            People

              marcuse Marcus Eriksson
              rstrickland Robert Strickland
              Marcus Eriksson
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: