Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-19092 ABFS phase 4: post Hadoop 3.4.0 features
  3. HADOOP-19120

[ABFS]: ApacheHttpClient adaptation as network library

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.5.0
    • None
    • fs/azure

    Description

      Apache HttpClient is more feature-rich and flexible and gives application more granular control over networking parameter.

      ABFS currently relies on the JDK-net library. This library is managed by OpenJDK and has no performance problem. However, it limits the application's control over networking, and there are very few APIs and hooks exposed that the application can use to get metrics, choose which and when a connection should be reused. ApacheHttpClient will give important hooks to fetch important metrics and control networking parameters.

      A custom implementation of connection-pool is used. The implementation is adapted from the JDK8 connection pooling. Reasons for doing it:
      1. PoolingHttpClientConnectionManager heuristic caches all the reusable connections it has created. JDK's implementation only caches limited number of connections. The limit is given by JVM system property "http.maxConnections". If there is no system-property, it defaults to 5. Connection-establishment latency increased with all the connections were cached. Hence, adapting the pooling heuristic of JDK netlib,
      2. In PoolingHttpClientConnectionManager, it expects the application to provide `setMaxPerRoute` and `setMaxTotal`, which the implementation uses as the total number of connections it can create. For application using ABFS, it is not feasible to provide a value in the initialisation of the connectionManager. JDK's implementation has no cap on the number of connections it can have opened on a moment. Hence, adapting the pooling heuristic of JDK netlib,

      Attachments

        Issue Links

          Activity

            People

              pranavsaxena Pranav Saxena
              pranavsaxena Pranav Saxena
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated: