Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-14187 FLIP-56 Dynamic Slot Allocation
  3. FLINK-17061

Unset process/flink memory size from configuration once dynamic worker resource is activated.

    XMLWordPrintableJSON

Details

    Description

      With FLINK-14106, memory of a TaskExecutor is decided in two steps on active resource managers.

      • SlotManager decides WorkerResourceSpec, including memory used by Flink tasks: task heap, task off-heap, network and managed memory.
      • ResourceManager derives TaskExecutorProcessSpec from WorkerResourceSpec and the configuration, deciding sizes of memory used by Flink framework and JVM: framework heap, framework off-heap, jvm metaspace and jvm overhead.

      This works fine for now, because both WorkerResourceSpec and TaskExecutorProcessSpec are derived from the same configurations. However, it might cause problem if later we have new SlotManager implementations that decides WorkerResourceSpec dynamically. In such cases, the process/flink sizes in configuration should be ignored, or it may easily lead to configuration conflicts.

      Attachments

        Issue Links

          Activity

            People

              xtsong Xintong Song
              xtsong Xintong Song
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: