Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-16561

Use autoSoftCommmitMaxTime as preferred poll interval of IndexFetcher

    XMLWordPrintableJSON

Details

    Description

      TLOG/PULL replicas use IndexFetcher to fetch segment files from leaders. Once new segment files are downloaded and merged into existing index, a new Searcher is opened so the updated data is made available to the clients.  The poll interval is determined by following code in ReplicateFromLeader

      if (uinfo.autoCommmitMaxTime != -1) {
         pollIntervalStr = toPollIntervalStr(uinfo.autoCommmitMaxTime/2);
      } else if (uinfo.autoSoftCommmitMaxTime != -1) {
         pollIntervalStr = toPollIntervalStr(uinfo.autoSoftCommmitMaxTime/2);
      }

       

      In a typical config for replication using TLOG/PULL replicas where data visibility is less important (a trade-off to avoid NRT replicas), we set a short commit time to persist changes and long soft-commit time to make changes visible.

       

      <autoCommit>
        <maxTime>15000</maxTime>
        <openSearcher>false</openSearcher>
      </autoCommit>
      <autoSoftCommit>
        <maxTime>3600000</maxTime>
      </autoSoftCommit>
      

       

      With about config, the poll interval will be 15/2 = 7 sec.  This leads to frequent opening of new Searchers which causes huge impact on realtime user queries, especially if the new Searcher takes long time to warmup.  This also makes changes visible on followers ahead of leaders.   

      Because the polling of new segment files is more about visibility because TLOG replicas still get updates to tlog files via UpdateHandler (this is my understanding). It seems more appropriate to use  autoSoftCommmitMaxTime as the poll interval.   

      I would  proposed change below where autoSoftCommmitMaxTime is chosen as the preferred interval.  This will make the poll interval much longer and make the visibility order more inline with eventual consistency pattern.

       

      if (uinfo.autoSoftCommmitMaxTime != -1) {
          pollIntervalStr = toPollIntervalStr(uinfo.autoSoftCommmitMaxTime);
      } else if (uinfo.autoCommmitMaxTime != -1) {
          pollIntervalStr = toPollIntervalStr(uinfo.autoCommmitMaxTime);
      }
      

      The change has been tried and showed much less impact on realtime queries.

      Attachments

        1. SOLR-16561.patch
          1 kB
          Hang Sun

        Issue Links

          Activity

            People

              elyograg Shawn Heisey
              sunh11373 Hang Sun
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 10m
                  10m