Improve error handling of remote queries (proxied requests)

The current remoteQuery code in SolrDispatchFilter yields error messages like the following:

org.apache.solr.servlet.SolrDispatchFilter: null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://localhost:8983/solr/myCollection/update
at org.apache.solr.servlet.SolrDispatchFilter.remoteQuery(SolrDispatchFilter.java:580)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:288)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:169)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.solr.servlet.ProxyUserFilter.doFilter(ProxyUserFilter.java:241)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.solr.servlet.SolrHadoopAuthenticationFilter$2.doFilter(SolrHadoopAuthenticationFilter.java:140)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:384)
at org.apache.solr.servlet.SolrHadoopAuthenticationFilter.doFilter(SolrHadoopAuthenticationFilter.java:145)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.solr.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.io.IOException: Server returned HTTP response code: 401 for URL: http://search-testing-c4-secure-4.ent.cloudera.com:8983/solr/sentryCollection/update?stream.body=%3Cadd%3E%3Cdoc%3E%3Cfield+name%3D%22id%22%3E1383855038349doc1%3C%2Ffield%3E%3Cfield+name%3D%22description%22%3Efirst+test+document+1383855038349%3C%2Ffield%3E%3C%2Fdoc%3E%3C%2Fadd%3E&doAs=user1
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1674)
at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1672)
at java.security.AccessController.doPrivileged(Native Method)
at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1670)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1243)
at org.apache.solr.servlet.SolrDispatchFilter.remoteQuery(SolrDispatchFilter.java:567)
... 25 more
Caused by: java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:8983/solr/myCollection/update?stream.body=%3Cadd%3E%3Cdoc%3E%3Cfield+name%3D%22id%22%3E1383855038349doc1%3C%2Ffield%3E%3Cfield+name%3D%22description%22%3Efirst+test+document+1383855038349%3C%2Ffield%3E%3C%2Fdoc%3E%3C%2Fadd%3E&doAs=user1
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1625)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
at org.apache.solr.servlet.SolrDispatchFilter.remoteQuery(SolrDispatchFilter.java:550)
... 25 more

In this case, the request handler threw an exception, and all the user got back was an error code, no message. They would actually have to dig through the logs on the remote machine to the see the error.

I tried for a bit to get the error message with HttpURLConnection, but wasn't successful. Instead, I used httpclient, like SolrServer uses. This works, as SolrServer already gives reasonable error messages.

This approach of using httpclient has another advantage as well: because the httpclient is created via the HttpClientUtil in the same way as the other http clients, any configuration settings are automatically picked up. For example, I have an HttpClientConfigurer that I wrote to handle kerberos connections; with this approach, the forwarded requests just work with kerberos. With the old approach, I would have to modify the remoteQuery code to do kerberos-specific things.