Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-16318

Memtable heap size is severely underestimated

Agile BoardAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Normal
    • Resolution: Fixed
    • 4.0-rc1, 4.0
    • Local/Memtable
    • None

    Description

      We seem to be estimating the size of the on-heap memtable metadata to be around half of what it actually is. For example, during a read benchmark which writes 1 million single-long rows the memtable reports

      1000000 ops, 58.174MiB serialized bytes, 385.284MiB (19%) on heap, 0.000KiB (0%) off-heap
      

      while a heap dump taken at this point:

      lists an usage of about 666MB altogether.

      Switching to offheap_objects, the reported numbers are

      1000000 ops, 58.174MiB serialized bytes, 233.650MiB (12%) on heap, 53.406MiB (3%) off-heap
      

      while actual heap usage:

      is about 442MB.

      Looking at the code we definitely are not counting the AtomicBTreePartition.Holder, EncodingStats, liveness and deletion info objects associated with each partition, and most probably others.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            e.dimitrova Ekaterina Dimitrova Assign to me
            blambov Branimir Lambov
            B Lambov (old handle), Ekaterina Dimitrova
            Benjamin Lerer, B Lambov (old handle)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment