Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-17870

nodetool/rebuild: Add flag to exclude nodes from local datacenter

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Normal
    • Resolution: Fixed
    • 5.0-alpha1, 5.0
    • Tool/nodetool
    • None

    Description

      During expansion by Dc, when we issue nodetool/rebuild from new dc to rebuild the data from other DCs. If src-dc is not passed explicitly, then C* tries to rebuild the data from the same (new dc) dc. 

      We don’t exclude other nodes in the same DC. Only down sources and the local node itself are excluded.
      ```
       // We're always filtering out a local node and down sources
              addSourceFilter(new RangeStreamer.FailureDetectorSourceFilter(failureDetector));
              addSourceFilter(new RangeStreamer.ExcludeLocalNodeFilter());
      ```

      We should fix nodetool/rebuild to exclude the local DC (from where we’re executing the command) while issuing nodetool/rebuild without passing src dc

       

      Example:
      in a 3 DC cluster,
      ks1 has DC1, DC2
      ks2 has DC1, DC2, DC3
      ks3 has DC2

      now, we add a new DC [DC4] and configured it to all 3 keyspaces.

      if we run rebuild with src DC as DC1, ks3 will fail as it does not have DC1.

      Now, without src DC, the expectation is rebuild would auto pick up DCs for each keyspace (let's say ks1: DC1, ks2: DC1, ks3: DC2) and would never fail due to under-replicated keyspaces.

      The issue with this approach (without src dc) is that, DC4 is getting picked up during rebuild (as src), but DC4 does not have any data yet!

      so, with the patch (ignore local dc flag), DC4 can be filtered out and let the database pick up the right dc for each keyspace [from existing 3 DCs].
      – this is what is the expectation after the patch.

      Attachments

        1. fix_nodetool_rebuild.diff
          9 kB
          Saranya Krishnakumar

        Activity

          People

            saranya_k Saranya Krishnakumar
            saranya_k Saranya Krishnakumar
            Saranya Krishnakumar
            Dinesh Joshi, Marcus Eriksson, Yifan Cai
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 1.5h
                1.5h