Uploaded image for project: 'Axis2'
  1. Axis2
  2. AXIS2-2593

Web Service request loop causes many TIME_WAIT Connections & "BindException: Address already in use: connect"

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 1.1.1, nightly
    • None
    • None
    • None
    • Windows XP, Tomcat, Axis2 nightly build (also encountered in 1.1.1), JiBX 1.1.3, Microsoft Windows XP [Version 5.1.2600], 126 GB free disk space, 1GB RAM.

    Description

      I am performing the following:

      • Executing a tight loop calling a singel web service
      • Using the same Stub over and over
      • Creating the Stub with a custom ConfigurationContext (see below)
      • Sending cleanup() to Stub after each message sent (although problem existed even before cleanup() was added)

      After approximately 4000 messages have been sent, an AxisFault is generated caused by the following exception (see below for full stack trace):

      java.net.BindException: Address already in use: connect

      It appears that connections from the client to Axis on Tomcat sometimes (but not always) remain in the TIME_WAIT state for several minutes. There are 1200-1500 TIME_WAIT connections open when the AxisFault is generated.

      If I step through the web service messages in a debugger, the problem does not appear to occur, as the system eventually releases the connections after a minute or two.

      Also, if I insert a System.gc() in the client code each time it sends a message to the web service (the commented out line of code below), the AxisFault does not occur. I have executed the loop over 60,000 times with the garbage collection included without generating an AxisFault. There were only 400-700 connections in the TIME_WAIT state at a given time, but performance slowed down to a crawl.

      Here is the code I am using to create the Stub:

      HttpConnectionManagerParams connectionManagerParams = new HttpConnectionManagerParams();
      connectionManagerParams.setTcpNoDelay(true);
      connectionManagerParams.setStaleCheckingEnabled(true);

      MultiThreadedHttpConnectionManager connectionManager = new MultiThreadedHttpConnectionManager();
      connectionManager.setParams(connectionManagerParams);
      HttpClient httpClient = new HttpClient(connectionManager);

      ConfigurationContext configurationContext = ConfigurationContextFactory.createConfigurationContextFromFileSystem(null, null);
      configurationContext.setProperty(HTTPConstants.REUSE_HTTP_CLIENT, Boolean.TRUE);
      configurationContext.setProperty(HTTPConstants.CACHED_HTTP_CLIENT, httpClient);

      stub = new MyServiceStub(configurationContext, target);

      boolean success = true;

      while(success)

      { success = stub.performService(records); stub.cleanup(); // System.gc(); }

      Here is the complete stack trace:

      Exception in thread "main" org.apache.axis2.AxisFault: Address already in use: connect
      at org.apache.axis2.AxisFault.makeFault(AxisFault.java:377)
      at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:179)
      at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:73)
      at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:310)
      at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:202)
      at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:446)
      at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:330)
      at org.apache.axis2.description.OutInAxisOperationClient.execute(OutInAxisOperation.java:294)
      at com.ws.service.MyServiceStub.performService(MyServiceStub.java:300)
      at com.ws.client.MyClient.main(MyClient.java:88)
      Caused by: java.net.BindException: Address already in use: connect
      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
      at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
      at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
      at java.net.Socket.connect(Socket.java:519)
      at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:139)
      at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:124)
      at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:706)
      at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:386)
      at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
      at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
      at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
      at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:558)
      at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:176)
      ... 8 more

      Attachments

        1. JIRA_test.tar.gz
          985 kB
          Michele Mazzucco
        2. diff.txt
          4 kB
          Davanum Srinivas
        3. JIRA-935_2593.zip
          990 kB
          Davanum Srinivas

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            dims Davanum Srinivas
            dbueche David Bueche
            Votes:
            2 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment