Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-4730

test-framework exits with SIGABRT

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Cannot Reproduce
    • None
    • None
    • None
    • OSX 10.11.3 (El Cap).

    Description

      Steps to repro:

      1. Build mesos from git
      2. ./src/mesos-master.sh --registry=in_memory
      3. ./src/mesos-slave.sh --master=X:5050
      4. ./src/test-framework --master=X:5050

      Observed behavior:

      $ ./src/test-framework --master=10.0.0.11:5050
      I0221 16:55:39.760979 1933606912 sched.cpp:222] Version: 0.28.0
      I0221 16:55:39.768154 2674688 sched.cpp:326] New master detected at master@10.0.0.11:5050
      I0221 16:55:39.768378 2674688 sched.cpp:336] No credentials provided. Attempting to register without authentication
      I0221 16:55:39.769650 2674688 sched.cpp:703] Framework registered with 227af8fe-56b7-4853-bc65-4076bd7be95d-0000
      Registered!
      Received offer 227af8fe-56b7-4853-bc65-4076bd7be95d-O0 with cpus(*):8; mem(*):15360; disk(*):233112; ports(*):[31000-32000]
      Launching task 0 using offer 227af8fe-56b7-4853-bc65-4076bd7be95d-O0
      Launching task 1 using offer 227af8fe-56b7-4853-bc65-4076bd7be95d-O0
      Launching task 2 using offer 227af8fe-56b7-4853-bc65-4076bd7be95d-O0
      Launching task 3 using offer 227af8fe-56b7-4853-bc65-4076bd7be95d-O0
      Launching task 4 using offer 227af8fe-56b7-4853-bc65-4076bd7be95d-O0
      Received offer 227af8fe-56b7-4853-bc65-4076bd7be95d-O1 with cpus(*):3; mem(*):14720; disk(*):233112; ports(*):[31000-32000]
      Task 0 is in state TASK_FAILED
      Aborting because task 0 is in unexpected state TASK_FAILED with reason 1 from source 1 with message 'Executor terminated'
      I0221 16:55:49.532826 528384 sched.cpp:1937] Asked to abort the driver
      I0221 16:55:49.532888 528384 sched.cpp:1173] Aborting framework '227af8fe-56b7-4853-bc65-4076bd7be95d-0000'
      I0221 16:55:49.533144 1933606912 sched.cpp:1903] Asked to stop the driver
      

      Content of agent's stderr file for the crashed executor:

      ABORT: (../../../mesos/3rdparty/libprocess/src/subprocess.cpp:322): Failed to os::execvpe on path '/usr/local/libexec/mesos/mesos-containerizer': No such file or directory
      *** Aborted at 1456102539 (unix time) try "date -d @1456102539" if you are using GNU date ***
      PC: @     0x7fff9666f002 __pthread_kill
      *** SIGABRT (@0x7fff9666f002) received by PID 2637 (TID 0x700000104000) stack trace: ***
          @     0x7fff926e6eaa _sigtramp
          @     0x7000000ff7e0 (unknown)
          @     0x7fff867d96e7 abort
          @        0x10e087010 _Abort()
          @        0x10e086e5b _Abort()
          @        0x1109a3938 process::childMain()
          @        0x1109b3269 _ZNSt3__128__invoke_void_return_wrapperIiE6__callIJRNS_6__bindIPFiRKNS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEEPPcSD_RK6OptionINS_8functionIFivEEEERKN7process10Subprocess2IO20InputFileDescriptorsERKNSN_21OutputFileDescriptorsEST_bPiEJSB_RSD_SX_SK_RSO_RSR_SZ_RKbRA2_iEEEEEEiDpOT_
          @        0x1109b267c _ZNSt3__110__function6__funcINS_6__bindIPFiRKNS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEEPPcSC_RK6OptionINS_8functionIFivEEEERKN7process10Subprocess2IO20InputFileDescriptorsERKNSM_21OutputFileDescriptorsESS_bPiEJSA_RSC_SW_SJ_RSN_RSQ_SY_RKbRA2_iEEENS6_IS13_EESF_EclEv
          @        0x10f815721 std::__1::function<>::operator()()
          @        0x1109a2f33 process::defaultClone()
          @        0x1109b0fdd _ZNSt3__128__invoke_void_return_wrapperIiE6__callIJRPFiRKNS_8functionIFivEEEES7_EEEiDpOT_
          @        0x1109b0e6c std::__1::__function::__func<>::operator()()
          @        0x1109ad3b7 std::__1::function<>::operator()()
          @        0x1109a0fd0 process::subprocess()
          @        0x10f814653 mesos::internal::slave::PosixLauncher::fork()
          @        0x10f658c7d mesos::internal::slave::MesosContainerizerProcess::__launch()::$_5::operator()()
          @        0x10f65d289 _ZZZNK7process9_DeferredIZN5mesos8internal5slave25MesosContainerizerProcess8__launchERKNS1_11ContainerIDERKNS1_12ExecutorInfoERKNSt3__112basic_stringIcNSB_11char_traitsIcEENSB_9allocatorIcEEEERK6OptionISH_ERKNS1_7SlaveIDERKNS_3PIDINS3_5SlaveEEEbRKNSB_4listISK_INS1_5slave19ContainerLaunchInfoEENSF_ISZ_EEEEE3$_5EcvNSB_8functionIFT_T0_EEEINS_6FutureIbEERKNSX_15ContainerLogger14SubprocessInfoEEEvENKUlS1H_E_clES1H_ENKUlvE_clEv
          @        0x10f65d254 _ZNSt3__128__invoke_void_return_wrapperIN7process6FutureIbEEE6__callIJRZZNKS1_9_DeferredIZN5mesos8internal5slave25MesosContainerizerProcess8__launchERKNS7_11ContainerIDERKNS7_12ExecutorInfoERKNS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEERK6OptionISM_ERKNS7_7SlaveIDERKNS1_3PIDINS9_5SlaveEEEbRKNS_4listISP_INS7_5slave19ContainerLaunchInfoEENSK_IS14_EEEEE3$_5EcvNS_8functionIFT_T0_EEEIS3_RKNS12_15ContainerLogger14SubprocessInfoEEEvENKUlS1K_E_clES1K_EUlvE_EEES3_DpOT_
          @        0x10f65cf83 _ZNSt3__110__function6__funcIZZNK7process9_DeferredIZN5mesos8internal5slave25MesosContainerizerProcess8__launchERKNS4_11ContainerIDERKNS4_12ExecutorInfoERKNS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEERK6OptionISJ_ERKNS4_7SlaveIDERKNS2_3PIDINS6_5SlaveEEEbRKNS_4listISM_INS4_5slave19ContainerLaunchInfoEENSH_IS11_EEEEE3$_5EcvNS_8functionIFT_T0_EEEINS2_6FutureIbEERKNSZ_15ContainerLogger14SubprocessInfoEEEvENKUlS1J_E_clES1J_EUlvE_NSH_IS1L_EEFS1F_vEEclEv
          @        0x10ec26510 std::__1::function<>::operator()()
          @        0x10ec26433 _ZZN7process8dispatchIbEENS_6FutureIT_EERKNS_4UPIDERKNSt3__18functionIFS3_vEEEENKUlPNS_11ProcessBaseEE_clESE_
          @        0x10ec263f0 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN7process8dispatchIbEENS3_6FutureIT_EERKNS3_4UPIDERKNS_8functionIFS7_vEEEEUlPNS3_11ProcessBaseEE_SH_EEEvDpOT_
          @        0x10ec2608c _ZNSt3__110__function6__funcIZN7process8dispatchIbEENS2_6FutureIT_EERKNS2_4UPIDERKNS_8functionIFS6_vEEEEUlPNS2_11ProcessBaseEE_NS_9allocatorISH_EEFvSG_EEclEOSG_
          @        0x1108c73a2 std::__1::function<>::operator()()
          @        0x1108a106f process::ProcessBase::visit()
          @        0x1108f411e process::DispatchEvent::visit()
          @        0x10e0cbf71 process::ProcessBase::serve()
          @        0x11089dd61 process::ProcessManager::resume()
          @        0x1108a961f process::ProcessManager::init_threads()::$_1::operator()()
          @        0x1108a92a2 _ZNSt3__114__thread_proxyINS_5tupleIJNS_6__bindIZN7process14ProcessManager12init_threadsEvE3$_1JNS_17reference_wrapperIKNS_6atomicIbEEEEEEEEEEEEPvSD_
          @     0x7fff90186c13 _pthread_body
          @     0x7fff90186b90 _pthread_start
      

      Attachments

        1. mesos-slave-console-log.txt
          12 kB
          Neil Conway
        2. mesos-master-console-log.txt
          18 kB
          Neil Conway

        Issue Links

          Activity

            People

              Unassigned Unassigned
              neilc Neil Conway
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: