XMLWordPrintableJSON

Details

    Description

      Currently, tcp connections can be only reuse by tasks residing in the same TaskManager and consumes the same IntermediateResult. And after job finish or failover, the TCP connections are closed and new connections must be setup latter.

      As an improvement, we can make tcp connections a cluster level resource which can be reused by multi jobs. The advantages are as follows:

      1. Reduce the number of TCP connections so we can save some resources.
      2. Reduce the overhead of connection setup and close so restarted jobs after failover and latter jobs submitted to the same session cluster can reuse the previous connections.

      We use Flink session cluster as a service for ad-hoc queries and the users can produce some statistics or create some statements and reports at any time. Most of the queries finish in 2s and we find tcp connection reuse help a lot to reduce the average execution time which means more queries can be processed using the same resource and time with even better user experience.

      Attachments

        Issue Links

          Activity

            People

              guoyangze Yangze Guo
              kevin.cyj Yingjie Cao
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: