Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-9501

Mesos executor fails to terminate and gets stuck after agent host reboot.

    XMLWordPrintableJSON

Details

    • Containerization R9 Sprint 37
    • 5

    Description

      When an agent host reboots, all of its containers are gone but the agent will still try to recover from its checkpointed state after reboot.

      The agent will soon discover that all the cgroup hierarchies are gone and assume (correctly) that the containers are destroyed.

      However, when trying to terminate the executor, the agent will first try to wait for the exit status of its container:
      https://github.com/apache/mesos/blob/master/src/slave/containerizer/mesos/containerizer.cpp#L2631

      Agent dose so by `waitpid` on the checkpointed child process pid. If, after the agent host reboot, a new process with the same pid gets spawned, then the parent will wait for the wrong child process. This could get stuck until the wrongly waited-for process is somehow exited, see `ReaperProcess::wait()`: https://github.com/apache/mesos/blob/master/3rdparty/libprocess/src/reap.cpp#L88-L114

      This will block the executor termination as well as future task status update (e.g. master might still think the task is running).

      Attachments

        Issue Links

          Activity

            People

              qianzhang Qian Zhang
              mzhu Meng Zhu
              Gilbert Song Gilbert Song
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: