Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-19013

Performance issues after Cassandra upgrade

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Normal
    • Resolution: Invalid
    • None
    • Local/Config
    • None
    • All
    • None

    Description

      I would like to ask for your help in the following issue.
       
      We have upgraded our Cassandra 3.11.4 to 4.1.2 in production environment and since that we're facing with many performance issues:

      • The cluster operations lowered to 17-20k/s which was between 21-25k/s with 3.11.4
      • CPU usage significantly increased -> previously it was nearly 50% at peak times, after the update it's between 65-70%
      • Pending tasks increased -> from nearly 0 to 30 at peak times
      • Native Transport Requests significantly increased -> from 100-200 to 2K-3K at peak times
      • Chunck cache hit rate metric stopped working
      • Read latency increased from microsecs to milisecs
      • sstables per read histogram also shows increased values

      We have already done with:

      • file_cache_enabled has been enabled
      • sstableupgrade
      • Java has been updated from 8 to 11
      • Made a truncate of affected keyspace
      • Affected keyspace's indexes has been rebuilt

      Our infrastructure contains 2 DCs with 24 nodes/dc. Every node has 8 CPUs and 64GBs of RAM and running on RHEL 8.6.

      We have already compared the two versions but didn't notice any new feature or option which could cause this performance degradation. If do you have any idea or do you need any kind of logs don't hesitate to contact me.

      Attachments

        Activity

          People

            Unassigned Unassigned
            dezsofim Márk Dezsőfi
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: