Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
The overseer queue is one of the busiest points in the entire system. The raison d'ĂȘtre of the queue is
- Provide batching of operations for the main clusterstate,json so that state updates are minimized
- Avoid race conditions and ensure order
Now , as we move the individual collection states out of the main clusterstate.json, the batching is not useful anymore.
Race conditions can easily be solved by using a compare and set in Zookeeper.
The proposed solution is , whenever an operation is required to be performed on the clusterstate, the same thread (and of course the same JVM)
- read the fresh state and version of zk node
- construct the new state
- perform a compare and set
- if compare and set fails go to step 1
This should be limited to all operations performed on external collections because batching would be required for others
Attachments
Issue Links
- relates to
-
SOLR-5475 Scale Overseer to work on multiple queues and threads
- Open
-
SOLR-10265 Overseer can become the bottleneck in very large clusters
- Open
-
SOLR-10524 Better ZkStateWriter batching
- Closed