Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-14940

ReplicationHandler memory leak through SolrCore.closeHooks

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 8.8, 9.0
    • replication (java)
    • None
    • Solr Cloud Cluster on v.8.6.2 configured as 3 TLOG nodes with 2 cores in each JVM.

       

    Description

      We are experiencing a memory leak in Solr Cloud cluster configured as 3 TLOG nodes.

      Leader does not seem to be affected while Followers are.

       

      Looking at memory dump we noticed that SolrCore holds lots of references to ReplicationHandler through anonymous inner classes in SolrCore.closeHooks, which in turn holds ReplicationHandlers.

      ReplicationHandler registers hooks as anonymous inner classes in SolrCore.closeHooks through ReplicationHandler.inform() -> ReplicationHandler.registerCloseHook().

       

      Whenever ZkController.stopReplicationFromLeader is called - it would shutdown ReplicationHandler (ReplicationHandler.shutdown()), BUT reference to ReplicationHandler will stay in SolrCore.closeHooks. Once replication is started again on same SolrCore - new ReplicationHandler will be created and registered in closeHooks.

       

      It looks like there are few scenarios when replication is stopped and restarted on same core and in our TLOG setup it shows up quite often.

       

      Potential solutions:

      1. Allow unregistering SolrCore.closeHooks so it can be used from ReplicationHandler.shutdown
      2. Hack but easier - break the link between ReplicationHandler close hooks and full ReplicationHandler object so ReplicationHandler can be GCed even when hooks are still registered in SolrCore.closeHooks

      Attachments

        Issue Links

          Activity

            People

              mdrob Mike Drob
              anverus Anver Sotnikov
              Votes:
              1 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h 10m
                  2h 10m