Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-15317

Parts of internal SolR communication fail when the CertAuthPlugin is active

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 8.8.1
    • 8.9, 8.8.3
    • Authentication
    • None
    • OS: CentOS 7

      Java: JDK 11

    Description

      Hi all!

      While I was testing out the CertAuthPlugin for the new SolR 9 it came to my attention that various internal HTTP calls in SolR fail. For example when I try to add a BinaryResponseWriter via curl it fails with lots of authentication errors (HTTP status code 401). Other actions (like creating schema fields for collections) via curl work fine. To reproduce the problem, following steps have to be taken (on Linux):

      • git clone https://github.com/apache/solr.git (I used commit caf8cbc0aa11e32f894a90531e3e9f20edf75efa)
      • cd solr
      • ./gradlew assemble
      • cd solr/packaging/build/solr-9.0.0-SNAPSHOT/
      • keytool -genkeypair -alias solr-ssl -keyalg RSA -keysize 2048 -keypass secret -storepass secret -validity 9999 -keystore solr-ssl.keystore.p12 -storetype PKCS12 -ext SAN=DNS:localhost,IP:127.0.0.1 -dname "CN=localhost, OU=Organizational Unit, O=Organization, L=Location, ST=State, C=Country"
      • openssl pkcs12 -in solr-ssl.keystore.p12 -out solr-ssl.keystore.key -nodes -nocerts
      • openssl pkcs12 -in solr-ssl.keystore.p12 -out solr-ssl.keystore.crt -nodes -nokeys
      • echo 'SOLR_SSL_ENABLED=true' >> bin/solr.in.sh
      • echo 'SOLR_SSL_KEY_STORE=../solr-ssl.keystore.p12' >> bin/solr.in.sh
      • echo 'SOLR_SSL_KEY_STORE_PASSWORD=secret' >> bin/solr.in.sh
      • echo 'SOLR_SSL_TRUST_STORE=../solr-ssl.keystore.p12' >> bin/solr.in.sh
      • echo 'SOLR_SSL_TRUST_STORE_PASSWORD=secret' >> bin/solr.in.sh
      • echo 'SOLR_SSL_NEED_CLIENT_AUTH=true' >> bin/solr.in.sh
      • echo 'SOLR_SSL_WANT_CLIENT_AUTH=false' >> bin/solr.in.sh
      • echo 'SOLR_SSL_CHECK_PEER_NAME=false' >> bin/solr.in.sh
      • ./bin/solr start -v -c
      • server/scripts/cloud-scripts/zkcli.sh -z localhost:9983 -cmd clusterprop -name urlScheme -val https
      • ./bin/solr zk cp file:///tmp/security.json zk:/security.json -z localhost:9983
      • ./bin/solr stop
      • ./bin/solr start -v -c
      • ./bin/solr create -c testcollection
      • curl --cacert ./solr-ssl.keystore.crt --key ./solr-ssl.keystore.key --cert ./solr-ssl.keystore.crt "https://localhost:8983/api/collections/testcollection/config" -H "Content-Type: application/json" --data-binary '{ "add-queryresponsewriter":{ "class":"solr.BinaryResponseWriter", "name":"test" }}'

       

      After the last curl command (which takes about 30 seconds) the following error message is printed:

      {{

      { "responseHeader":

      { "status":500, "QTime":30017}

      , "errorMessages":["1 out of 2 the property overlay to be of version 0 within 30 seconds! Failed cores: https://localhost:8983/solr/testcollection_shard1_replica_n1/\n"], "WARNING":"This response format is experimental. It is likely to change in the future.", "error":{ "metadata":[ "error-class","org.apache.solr.common.SolrException", "root-error-class","org.apache.solr.common.SolrException"], "msg":"1 out of 2 the property overlay to be of version 0 within 30 seconds! Failed cores: https://localhost:8983/solr/testcollection_shard1_replica_n1/", "trace":"org.apache.solr.common.SolrException: 1 out of 2 the property overlay to be of version 0 within 30 seconds! Failed cores: https://localhost:8983/solr/testcollection_shard1_replica_n1/\n\tat org.apache.solr.handler.SolrConfigHandler.waitForAllReplicasState(SolrConfigHandler.java:829)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handleCommands(SolrConfigHandler.java:549)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handlePOST(SolrConfigHandler.java:381)\n\tat org.apache.solr.handler.SolrConfigHandler.handleRequestBody(SolrConfigHandler.java:140 )\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:214)\n\tat org.apache.solr.api.ApiBag$ReqHandlerToApi.call(ApiBag.java:269)\n\tat org.apache.solr.api.V2HttpCall.execute(V2HttpCall.java:354)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:567)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:518)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:432)\n\tat org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)\n\tat org.eclipse.jetty.servlet. ServletHandler$Chain.doFilter(ServletHandler.java:1601)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1612)\n\tat org.eclipse.j etty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1582)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:134 9)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)\n\tat org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)\n\tat org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)\n\tat org.eclipse.jetty.server.hand ler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:516)\n\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)\n\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.jav a:105)\n\tat org.eclipse.jetty.io.ssl.SslConnection$1.run(SslConnection.java:146)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n", "code":500}}}}

       

      In the SolR WEB-UI lots of PKIAuthentication errors are printed (see solr_error.png). Out of curiosity I backported the CertAuthPlugin to SolR v8.8.1 locally and SolR 8 had the same issues as the current master.

      Attachments

        1. security.json
          0.5 kB
          Dominik Dresel
        2. solr_error.png
          105 kB
          Dominik Dresel
        3. solr-snippet.log.bz2
          452 kB
          Dominik Dresel

        Issue Links

          Activity

            People

              mdrob Mike Drob
              dreseldo Dominik Dresel
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m