Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-13184

Starting a TaskExecutor blocks the YarnResourceManager's main thread

    XMLWordPrintableJSON

Details

    Description

      Currently, YarnResourceManager starts all task executors in main thread. This could cause RM to become unresponsive when launching a large number of TEs (e.g. > 1000) because it involves blocking I/O operations (writing files to HDFS, communicating with the node manager using a synchronous NMClient). As a consequence, TE registration/heartbeat timeouts can occur and Flink might allocate too many excessive containers (see FLINK-12342) because it cannot process the YarnResourceManager#onContainersAllocated calls.

      There are different solution approaches but the end goal should be to not execute any blocking calls in the ResourceManager's main thread:

      1. Start the TaskExecutors from a different thread (potentially thread pool) which is responsible for uploading the files and communicating with the NodeManager
      2. Don't upload files (avoid blocking file system operations) and use the NMClientAsync for the communication with Yarn's NodeManager.
      3. Upload files in a separate I/O thread and use the NMClientAsync for the communication with Yarn's NodeManager.

      Attachments

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. FLINK-12342 Yarn Resource Manager Acquires Too Many Containers

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. FLINK-13590 flink-on-yarn sometimes could create many little files that are xxx-taskmanager-conf.yaml

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-9410 Replace NMClient with NMClientAsync in YarnResourceManager

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-14582 Do not upload {uuid}-taskmanager-conf.yaml for each task manager container

          • Major - Major loss of function.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. FLINK-15053 Configurations with values contains space may cause TM failures on Yarn

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          links to

          Web Link GitHub Pull Request #10143

          Web Link GitHub Pull Request #10259

          Web Link GitHub Pull Request #10260

          Activity

            People

              wangyang0918 Yang Wang
              xtsong Xintong Song
              Votes:
              2 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 20m
                  1h 20m