Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-14654

Reduce heap pressure during compactions

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Hide

      TBD

      Show
      TBD

    Description

      Small partition compactions are painfully slow with a lot of overhead per partition. There also tends to be an excess of objects created (ie 200-700mb/s) per compaction thread.

      The EncoderStats walks through all the partitions and with mergeWith it will create a new one per partition as it walks the potentially millions of partitions. In a test scenario of about 600byte partitions and a couple 100mb of data this consumed ~16% of the heap pressure. Changing this to instead mutably track the min values and create one in a EncodingStats.Collector brought this down considerably (but not 100% since the UnfilteredRowIterator.stats() still creates 1 per partition).

      The KeyCacheKey makes a full copy of the underlying byte array in ByteBufferUtil.getArray in its constructor. This is the dominating heap pressure as there are more sstables. By changing this to just keeping the original it completely eliminates the current dominator of the compactions and also improves read performance.

      Minor tweak included for this as well for operators when compactions are behind on low read clusters is to make the preemptive opening setting a hotprop.

      Attachments

        1. screenshot-1.png
          173 kB
          Chris Lohfink
        2. screenshot-2.png
          158 kB
          Chris Lohfink
        3. screenshot-3.png
          12 kB
          Chris Lohfink
        4. screenshot-4.png
          11 kB
          Chris Lohfink

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            cnlwsu Chris Lohfink Assign to me
            cnlwsu Chris Lohfink
            Chris Lohfink
            Benedict Elliott Smith, Dinesh Joshi
            Votes:
            1 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - Not Specified
              Not Specified
              Remaining:
              Remaining Estimate - 0h
              0h
              Logged:
              Time Spent - 50m
              50m

              Slack

                Issue deployment