Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-5872

Eliminate overseer queue

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • SolrCloud
    • None

    Description

      The overseer queue is one of the busiest points in the entire system. The raison d'ĂȘtre of the queue is

      • Provide batching of operations for the main clusterstate,json so that state updates are minimized
      • Avoid race conditions and ensure order

      Now , as we move the individual collection states out of the main clusterstate.json, the batching is not useful anymore.

      Race conditions can easily be solved by using a compare and set in Zookeeper.

      The proposed solution is , whenever an operation is required to be performed on the clusterstate, the same thread (and of course the same JVM)

      1. read the fresh state and version of zk node
      2. construct the new state
      3. perform a compare and set
      4. if compare and set fails go to step 1

      This should be limited to all operations performed on external collections because batching would be required for others

      Attachments

        Issue Links

          Activity

            People

              noble.paul Noble Paul
              noble.paul Noble Paul
              Votes:
              2 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated: