Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-3627

Memory leak when using spellchecker with long function query (fq)

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Cannot Reproduce
    • 3.1, 3.6, 4.0-ALPHA
    • None
    • search, spellchecker
    • Windows 7 x64
      Tomcat 6.0.26
      Solr 3.1
      Java options: -XX:PermSize=128m -XX:MaxPermSize=128m -Xms1024m -Xmx1024m

    Description

      If spellcheck=true and request to Solr contains too long 'fq', memory is not cleaned, i.e. garbage collector can't clean memory. Fill rate of memory depends on the length of the fq.
      Example request:

      rows=10
      version=2.2
      spellcheck=true
      q=+((shortDesc_en:(hghg)))
      fq=(+contractIds:("contractId1" "contractId2" ... "contractId20000") AND +catalogIds:("catalogId1" "catalogId2" ... "catalogId20000"))
      

      solrconfig.xml:

      ...
      <requestHandler name="standard" class="solr.SearchHandler" default="true">
      	<!-- default values for query parameters -->
      	<lst name="defaults">
      		<str name="echoParams">explicit</str>
      
      		<str name="spellcheck.onlyMorePopular">true</str>
      		
      		<str name="spellcheck.extendedResults">false</str>
      		<str name="spellcheck.collate">true</str>
      		<str name="spellcheck.collateExtendedResults">true</str>
      		
      		<str name="spellcheck.maxCollations">3</str>
      		
      		<str name="spellcheck.maxCollationTries">6</str>
      		
      		<str name="spellcheck.count">3</str>
      	</lst>
      	<arr name="last-components">
      	        <str>spellcheck</str>
      	</arr>
      </requestHandler>
      ...
      
      If spellcheck.collate=false, issue is not reproduced.

      Preliminary research gave following result:
      When contractIds=~20000 and catalogIds=~20000, fill rate of memory is about 60 request.
      When contractIds=~10000 and catalogIds=~10000, fill rate of memory is about 90 request.

      Attachments

        1. test-query.txt
          606 kB
          Eugene Viktorovich
        2. solr-perfomance-test(2.5.1).jmx
          1009 kB
          Eugene Viktorovich
        3. solrconfig.xml
          64 kB
          Eugene Viktorovich
        4. SOLR-3627.patch
          0.8 kB
          Sergei Tkachuk
        5. params.txt
          606 kB
          Eugene Viktorovich
        6. jprofiler_analysis.zip
          8 kB
          Sergei Tkachuk
        7. jmeter-screen.png
          94 kB
          Eugene Viktorovich
        8. javaVisualVM-screen.png
          145 kB
          Eugene Viktorovich

        Issue Links

          Activity

            People

              Unassigned Unassigned
              volfgang Eugene Viktorovich
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: