Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-9291

Too many tombstones in schema_columns from creating too many CFs

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Urgent
    • Resolution: Duplicate
    • None
    • None
    • None
    • Critical

    Description

      When creating lots of columnfamilies (about 200) the system.schema_columns gets filled with tombstones and therefore prevents clients using the binary protocol of connecting.

      Clients already connected continue normal operation (reading and inserting).

      Log messages are:

      For the first tries (sorry for the lack of precision):

      ERROR [main] 2015-04-22 00:01:38,527 SliceQueryFilter.java (line 200) Scanned over 100000 tombstones in system.schema_columns; query aborted (see tombstone_failure_threshold)

      For each client that tries to connect but fails with timeout:

      WARN [ReadStage:35] 2015-04-27 15:40:10,600 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-]

      WARN [ReadStage:40] 2015-04-27 15:40:10,609 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-]

      WARN [ReadStage:61] 2015-04-27 15:40:10,670 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-]

      WARN [ReadStage:51] 2015-04-27 15:40:10,670 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-]

      WARN [ReadStage:55] 2015-04-27 15:40:10,675 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-]

      WARN [ReadStage:35] 2015-04-27 15:40:10,707 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-]

      WARN [ReadStage:40] 2015-04-27 15:40:10,708 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-]

      WARN [ReadStage:43] 2015-04-27 15:40:10,715 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-]

      WARN [ReadStage:51] 2015-04-27 15:40:10,736 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-]

      WARN [ReadStage:61] 2015-04-27 15:40:10,736 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-]

      WARN [ReadStage:35] 2015-04-27 15:40:10,750 SliceQueryFilter.java (line 231) Read 864 live and 2664 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147281748 columns was requested, slices=[-]

      WARN [ReadStage:40] 2015-04-27 15:40:10,751 SliceQueryFilter.java (line 231) Read 864 live and 2664 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147281748 columns was requested, slices=[-]

      WARN [ReadStage:55] 2015-04-27 15:40:10,759 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-]

      WARN [ReadStage:51] 2015-04-27 15:40:10,821 SliceQueryFilter.java (line 231) Read 864 live and 2664 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147281748 columns was requested, slices=[-]

      WARN [ReadStage:61] 2015-04-27 15:40:10,822 SliceQueryFilter.java (line 231) Read 864 live and 2664 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147281748 columns was requested, slices=[-]

      WARN [ReadStage:43] 2015-04-27 15:40:10,827 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-]

      WARN [ReadStage:55] 2015-04-27 15:40:10,838 SliceQueryFilter.java (line 231) Read 864 live and 2664 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147281748 columns was requested, slices=[-]

      WARN [ReadStage:62] 2015-04-27 15:40:10,846 SliceQueryFilter.java (line 231) Read 395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147283441 columns was requested, slices=[-]

      WARN [ReadStage:43] 2015-04-27 15:40:10,862 SliceQueryFilter.java (line 231) Read 864 live and 2664 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147281748 columns was requested, slices=[-]

      WARN [ReadStage:62] 2015-04-27 15:40:10,898 SliceQueryFilter.java (line 231) Read 1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147282894 columns was requested, slices=[-]

      WARN [ReadStage:62] 2015-04-27 15:40:10,970 SliceQueryFilter.java (line 231) Read 864 live and 2664 tombstoned cells in system.schema_columns (see tombstone_warn_threshold). 2147281748 columns was requested, slices=[-]

      This happens independently of values in:
      tombstone_warn_threshold, tombstone_failure_threshold

      Thrift doesn't seem to be vulnerable to this, as cqlsh connects every time.

      Binary protocol tested with Java (and Clojure), Python, C++ all fail to connect.

      Included in attachment are a cql script to replicate (with 500 CFs) and a sstablemetadata output of a system.schema_columns sstable before (clean cluster) and after importing the schema.

      Attachments

        1. after_schema.txt
          9 kB
          Luis Correia
        2. before_schema.txt
          21 kB
          Luis Correia
        3. schemas500.cql
          51 kB
          Luis Correia

        Issue Links

          Activity

            People

              Unassigned Unassigned
              luiscorreia Luis Correia
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: