Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-13556

Corrupted SSTables

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Normal
    • Resolution: Unresolved
    • 3.11.x
    • Local/Compaction
    • None
    • CentOS Linux release 7.3.1611 (Core)
      openjdk version "1.8.0_121"
      OpenJDK Runtime Environment (build 1.8.0_121-b13)
      OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)
      Python cassandra (DataStax) driver v3.6.0

    • Normal

    Description

      After 3 month of working, we noticed that number of compaction tasks were growing (~600 pending tasks). SStables verification shows that some of them were corrupted. Repairing didn't help (it was crashing with error).
      Also some of requests (f.e. select * from fetcher where domain=8289511971670945261 and uri=-5417197141545933706; ) fails with next error:

      Traceback (most recent call last):
      File "/var/cassandra/apache-cassandra-3.9/bin/cqlsh.py", line 1264, in perform_simple_statement
      result = future.result()
      File "/var/cassandra/apache-cassandra-3.9/bin/../lib/cassandra-driver-internal-only-3.5.0.post0-d8d0456.zip/cassandra-driver-3.5.0.post0-d8d0456/cassandra/cluster.py", line 3650, in result
      raise self._final_exception
      error: unpack requires a string argument of length 4

      Table chema:

      CREATE TABLE fetcher (
      domain bigint,
      uri bigint,
      date date,
      content_length int,
      elapsed float,
      encoding text,
      fetched_time bigint,
      flinks frozen<set<tuple<bigint, bigint»>,
      flinks_count int,
      html_fingerprint bigint,
      indexed boolean,
      adult boolean,
      kws_count int,
      lang_id int,
      last_updated bigint,
      redirect_url tuple<bigint, bigint>,
      revisit_date date,
      revisit_interval int,
      status_code int,
      tokens_fingerprint bigint,
      uris frozen<set<bigint»,
      url text,
      PRIMARY KEY (domain, uri, date)
      ) WITH CLUSTERING ORDER BY (uri ASC, date DESC)
      AND bloom_filter_fp_chance = 0.1
      AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
      AND comment = 'fetcher history'
      AND compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy',
      'sstable_size_in_mb': '256',
      'tombstone_threshold': '.2'}
      AND compression = {'chunk_length_in_kb': '64',
      'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
      AND crc_check_chance = 1.0
      AND dclocal_read_repair_chance = 0.1
      AND default_time_to_live = 0
      AND gc_grace_seconds = 864000
      AND max_index_interval = 2048
      AND memtable_flush_period_in_ms = 0
      AND min_index_interval = 128
      AND read_repair_chance = 0.5
      AND speculative_retry = '99PERCENTILE';

      Corrupted SSTable.

      Attachments

        Activity

          People

            Unassigned Unassigned
            prokopov94 Ihor Prokopov
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated: