Client port unavailable after binding the same client port during reconfig

When dynamically replace a server with IPv4/IPv6 with the same port, the server will complain about 'address already in use', and cause the client port not available anymore.