Delete collection time out after 180s or create the configset time out:300s

I have 1 zookeeper node and one solr node. Sometimes when trying to create or delete collection there is
"SEVERE: null:org.apache.solr.common.SolrException: delete the collection time out:180s"
Or "create the configset time out:300s.". This happens always until I restart solr. then after some time, it happens again until restart.

After checking the code I found that solr puts a task node to zookeeper /overseer/collection-queue-work/qnr-0000012764 /overseer/collection-queue-work/qn-0000012764 then a watcher listen for this and process the task, then delete the response node which triggers the latchWatcher to notify the thread that the task finished. The timeout for this is 180 seconds (hard coded). I think that sometimes the watcher to trigger the processor not triggered? Is that a bug? How to fix that?
Please help.