Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-9267

Mesos agent crashes when CNI network is not configured but used.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.4.2, 1.5.1, 1.6.1, 1.7.0
    • 1.5.2, 1.6.2, 1.7.1
    • None
    • None

    • Mesos 1.7.0

      Marathon 1.7.111

      Ubuntu 16.04

    Description

      I'm running Mesos 1.7.0 with Marathon 1.7.111 using the Mesos Universal Container Runtime to manage a Docker container. When I set the network mode to "container/bridge" the executor crashing instantly whenever the task is scheduled.

      Here are the syslogs

       

      Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]: I0926 16:49:58.604806 46045 containerizer.cpp:3116] Transitioning the state of container a29fd74a-a361-4b3e-8763-5f2cef77380d from PROVISIONING to PREPARING

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]: mesos-agent: ../../3rdparty/stout/include/stout/option.hpp:118: const T& Option<T>::get() const & [with T = std::__cxx11::basic_string<char>]: Assertion `isSome()' failed.

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]: *** Aborted at 1537980598 (unix time) try "date -d @1537980598" if you are using GNU date ***

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]: PC: @     0x7f29695df428 (unknown)

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]: *** SIGABRT (@0xb3d1) received by PID 46033 (TID 0x7f295b0ee700) from PID 46033; stack trace: ***

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f2969985390 (unknown)

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f29695df428 (unknown)

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f29695e102a (unknown)

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f29695d7bd7 (unknown)

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f29695d7c82 (unknown)

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f296c63c964 mesos::internal::slave::NetworkCniIsolatorProcess::getNetworkConfigJSON()

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f296c63d3e5 mesos::internal::slave::NetworkCniIsolatorProcess::prepare()

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f296c3700c6 _ZNO6lambda12CallableOnceIFvPN7process11ProcessBaseEEE10CallableFnINS_8internal7PartialIZNS1_8dispatchI6OptionIN5mesos5slave19ContainerLaunchInfoEENSB_8internal5slave20MesosIsolatorProcessERKNSB_11ContainerIDERKNSC_15ContainerConfigESK_SN_EENS1_6FutureIT_EERKNS1_3PIDIT0_EEMSS_FSQ_T1_T2_EOT3_OT4_EUlSt10unique_ptrINS1_7PromiseISE_EESt14default_deleteIS16_EEOSI_OSL_S3_E_IS19_SI_SL_St12_PlaceholderILi1EEEEEEclEOS3_

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f296ce67da1 process::ProcessBase::consume()

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f296ce89efa process::ProcessManager::resume()

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f296ce8dc46 _ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vEEE6_M_runEv

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f2969e5fc80 (unknown)

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f296997b6ba start_thread

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f29696b141d (unknown)

Sep 26 16:49:58 ip-172-27-1-88 systemd[1]: mesos-agent.service: Main process exited, code=killed, status=6/ABRT

Sep 26 16:49:58 ip-172-27-1-88 systemd[1]: mesos-agent.service: Unit entered failed state.

Sep 26 16:49:58 ip-172-27-1-88 systemd[1]: mesos-agent.service: Failed with result 'signal'.

Sep 26 16:49:58 ip-172-27-1-88 systemd[1]: mesos-agent.service: Service hold-off time over, scheduling restart.

Sep 26 16:49:58 ip-172-27-1-88 systemd[1]: Stopped Mesos Agent Service.

Sep 26 16:49:58 ip-172-27-1-88 systemd[1]: Started Mesos Agent Service.

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46255]: I0926 16:49:58.947785 46255 main.cpp:349] Build: 2018-09-21 14:54:37 by ubuntu

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46255]: I0926 16:49:58.947849 46255 main.cpp:350] Version: 1.7.0

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46255]: I0926 16:49:58.947855 46255 main.cpp:353] Git tag: 1.7.0

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46255]: I0926 16:49:58.947860 46255 main.cpp:357] Git SHA: 8419b870c571ac11825c883fa20ea3b7d4348d34

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46255]: I0926 16:49:58.952015 46255 systemd.cpp:240] systemd version `229` detected

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46255]: I0926 16:49:58.952047 46255 main.cpp:452] Initializing systemd state

Sep 26 16:49:58 ip-172-27-1-88 systemd[1]: Created slice Mesos Executors Slice.
Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46255]: I0926 16:49:58.955677 46255 systemd.cpp:328] Started systemd slice `mesos_executors.slice`

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46255]: I0926 16:49:58.956717 46255 resolver.cpp:69] Creating default secret resolver

       

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MESOS-8972 when choose docker image use user network all mesos agent crash

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. MESOS-8769 Agent crashes when CNI config not defined

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Activity

            People

              jieyu Jie Yu
              dkjs z s
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: