Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-7842

ZK connection loss or session expiry events should not fire config directory listeners

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      The watcher on the config directory has the following in the process method:

      Stat stat = null;
            try {
              stat = zkClient.exists(zkDir, null, true);
            } catch (KeeperException e) {
              //ignore , it is not a big deal
            } catch (InterruptedException e) {
              Thread.currentThread().interrupt();
            }
      
            boolean resetWatcher = false;
            try {
              resetWatcher = fireEventListeners(zkDir);
            } finally {
              if (Event.EventType.None.equals(event.getType())) {
                log.info("A node got unwatched for {}", zkDir);
              } else {
                if (resetWatcher) setConfWatcher(zkDir, this, stat);
                else log.info("A node got unwatched for {}", zkDir);
              }
            }
      

      Even if the watcher is fired because of session expiry or connection loss, the fireEventListeners() method is executed and all subsequent listener invocations fail due to the loss of connection/session. All this is logged as well.

      466879 WARN  (Thread-78) [   ] o.a.s.c.ZkController listener throws error
      org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /configs/jepsen/params.json
              at org.apache.solr.core.RequestParams.getFreshRequestParams(RequestParams.java:158)
              at org.apache.solr.core.SolrConfig.refreshRequestParams(SolrConfig.java:909)
              at org.apache.solr.core.SolrCore$11.run(SolrCore.java:2585)
              at org.apache.solr.cloud.ZkController$4.run(ZkController.java:2385)
      Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /configs/jepsen/params.json
              at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
              at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
              at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
              at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:302)
              at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:299)
              at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
              at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:299)
              at org.apache.solr.core.RequestParams.getFreshRequestParams(RequestParams.java:148)
              ... 3 more
      

      We should check the keeper state in addition to the event type and ignore such events.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            shalin Shalin Shekhar Mangar
            shalin Shalin Shekhar Mangar
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment