Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-34728

operator does not need to upload and download the jar when deploying session job

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • kubernetes-operator-1.5.0, kubernetes-operator-1.6.0, kubernetes-operator-1.7.0
    • None
    • None

    Description

      Problem:

      By reading the source code of the sessionjob's first reconcilition in the session mode of the flink kubernetes operator, a clear single point of bottleneck can be identified. When submitting a session job, the operator needs to first download the job jar from the jarURL to the local storage of kubernetes pod , then upload the jar to the job manager through the `/jars/upload` rest api, and finally call the `/jars/:jarid/run` rest api to launch the job.

      In this process, the operator needs to first download the jar and then upload the jar. When multiple jobs are submitted to the session cluster simultaneously, the operator can become a single point of bottleneck, which may be limited by the network traffic or other resource constraints of the operator pod.

       

      https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java#L824

       

      Solution:

      We can modify the job submission process in the session mode. The jobmanager can provide a `/jars/run` rest api that supports self-downloading the job jar, and the operator only needs to send a rest request to submit the job, without download and upload the job jar. In this way, the submission pressure of the operator can be distributed to each job manager. 

      Attachments

        Activity

          People

            Unassigned Unassigned
            Fei Feng Fei Feng
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: