Invalid hl URL params causes all future queries using hl to fail with "TokenStream contract violation: close() call missing"

We're using Solr 4.10.4 and came across an interesting find recently.

We're not certain if this issue exists in versions prior to 4.10.4 or newer 5.x and 6.x releases, but it's serious enough that when triggered, it can cause an entire website that is powered by Solr to stop functioning if highlighter is used.

Super simple schema.xml:
<?xml version="1.0" encoding="UTF-8" ?>
<schema name="sv_solr_v1_en" version="1.5">
<field name="version" type="long" indexed="true" stored="true" />
<field name="id" type="string" indexed="true" stored="true" required="true" />
<uniqueKey>id</uniqueKey>
<fieldType name="string" class="solr.StrField" sortMissingLast="true" />
<fieldType name="long" class="solr.TrieLongField" precisionStep="0" positionIncrementGap="0"/>
</schema>

To reproduce, the Solr core needs to have at least one document that matches the query. I inserted this test document using Solr admin for testing:

First request (hl.encoder not valid):
http://192.168.50.6:8983/solr/client_rc/select?q=*:*&wt=json&hl=true&hl.fl=id&hl.encoder=anything_invalid

First request (hl.fragsize non-numeric):
http://192.168.50.6:8983/solr/client_rc/select?q=*:*&wt=json&hl=true&hl.fl=id&hl.fragsize=123a

All requests fail after either of the above hl.encoder or hl.fragsize errors occur: http://192.168.50.6:8983/solr/client_rc/select?q=*:*&wt=json&hl=true&hl.fl=id

After finding this, we tested many other parameters outside/inside of the "hl" namespace, but it seems that only hl.encoder and hl.fragsize cause the problem. The first request gives you back the proper validation error.

*However, all future requests that use &hl=true&hl.fl=any_stored_field are stuck in the ​TokenStream contract violation: close() error state.*

It only affects the single Solr core, and doesn't permanently corrupt the index, just something is racing in the running Solr application it looks like from some error state not being properly handled.

To recover, you can UNLOAD + CREATE the core, or to RELOAD using Solr admin.

—

Full stack trace for the error:
​​​​java.lang.IllegalStateException: TokenStream contract violation: close() call missing
at org.apache.lucene.analysis.Tokenizer.setReader(Tokenizer.java:90)
at org.apache.lucene.analysis.Analyzer$TokenStreamComponents.setReader(Analyzer.java:323)
at org.apache.lucene.analysis.Analyzer.tokenStream(Analyzer.java:185)
at org.apache.solr.highlight.DefaultSolrHighlighter.createAnalyzerTStream(DefaultSolrHighlighter.java:642)
at org.apache.solr.highlight.DefaultSolrHighlighter.doHighlightingByHighlighter(DefaultSolrHighlighter.java:494)
at org.apache.solr.highlight.DefaultSolrHighlighter.doHighlighting(DefaultSolrHighlighter.java:414)
at org.apache.solr.handler.component.HighlightComponent.process(HighlightComponent.java:144)
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:218)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1976)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)

​---

After it's been in this bad state for a while, we start to see seemingly unrelated errors like this during indexing, which prevents index updates from completing. I'm not sure how they are related, but it only appears to happen during the bugged state:

org.apache.solr.common.SolrException: Exception writing document id events_23431 to the index; possible analysis error.
at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:168)
at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:926)
at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1080)
at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:692)
at org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.handleAdds(JsonLoader.java:460)
at org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.processUpdate(JsonLoader.java:132)
at org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.load(JsonLoader.java:106)
at org.apache.solr.handler.loader.JsonLoader.load(JsonLoader.java:68)

—

For now, we'll look into performing some pre-validation of hl.fragsize and hl.encoder on our end before connecting to Solr. For better or worse, uur applications currently pass-through many arguments directly to Solr, so this is a pretty big concern for us to resolve.

I also sent this to the general security mailing list, but with a different subject line "Lucene/Solr - Able to corrupt Solr instance via simple malformed URL query parameters"