Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-1514

A tablet that ends up under replicated will spam logs

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • consensus, supportability

    Description

      Trying to debug a tablet that got down to 1 replica is hard when these lines are printed multiple times per second:

      W0704 23:05:30.999037   312 transaction_tracker.cc:112] Transaction failed, tablet 807ff8e42640482d8d947b693d56ce03 transaction memory consumption (67107918) has exceeded its limit (67108864) or the limit of an ancestral tracker [suppressed 140 similar messages]
      I0704 23:05:31.000737 24321 consensus_peers.cc:181] T 807ff8e42640482d8d947b693d56ce03 P 9e59a4c24de44e3f9de219df865b4f3b -> Peer 94051c9253f94dadbc1af38098b41077 (e1105.halxg.cloudera.com:7050): Could not obtain request from queue for peer: 94051c9253f94dadbc1af38098b41077. Status: Not found: Failed to read ops 2302557..2325361: Segment 1118 which contained index 2302557 has been GCed
      I0704 23:05:31.000780 24452 raft_consensus.cc:629] T 807ff8e42640482d8d947b693d56ce03 P 9e59a4c24de44e3f9de219df865b4f3b [term 29 LEADER]: Processing failure of peer 94051c9253f94dadbc1af38098b41077 in term 29 (The logs necessary to catch up peer 94051c9253f94dadbc1af38098b41077 have been garbage collected. The follower will never be able to catch up (Not found: Failed to read ops 2302557..2325361: Segment 1118 which contained index 2302557 has been GCed)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
      I0704 23:05:31.138310   378 raft_consensus.cc:1603] T 807ff8e42640482d8d947b693d56ce03 P 9e59a4c24de44e3f9de219df865b4f3b [term 29 LEADER]: Leader election vote request: Denying vote to candidate 94051c9253f94dadbc1af38098b41077 for term 5380 because replica is either leader or believes a valid leader to be alive.
      

      Attachments

        Activity

          People

            Unassigned Unassigned
            jdcryans Jean-Daniel Cryans
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: