Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-8065

Implicitly declining offers in ACCEPT call as the launch operation specified no tasks - for GPU based job

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Invalid
    • 1.4.0
    • None
    • docker, framework, gpu
    • Ubuntu 14.04

    Description

      I tried changing the "docker-no-executor-framework" to require GPU resource and use mesos containerizer with a docker image of nvidia/cuda.

      This worked for me when i ran this framework with the master IP of one installed with DC/OS. But when I manually compiled Mesos (1.4/1.5) and started agent/master on another machine I face this issue : "Implicitly declining offers in ACCEPT call as the launch operation specified no tasks."
      I've tested the nvidia/cuda image to work on that target machine using nvidia-docker.

      Configuration:
      Master -
      ./bin/mesos-master.sh pas-lab-server6:5050 --work_dir=/home/sayantan/mesos_temp --filter_gpu_resources=false

      Agent (on the same node) -
      ./bin/mesos-agent.sh --master=pas-lab-server6:5050 --work_dir=/home/sayantan/mesos_temp --isolation="filesystem/linux,cgroups/devices,gpu/nvidia"

      I1010 14:48:44.106006 20372 sched.cpp:336] New master detected at master@127.0.1.1:5050
      I1010 14:48:44.107053 20372 sched.cpp:351] No credentials provided. Attempting to register without authentication
      I1010 14:48:44.113246 20365 sched.cpp:747] Framework registered with 00afc79a-cb51-4f9d-8c78-6000253ae396-0000
      Registered!
      .Starting task 0 on pas-lab-server6
      Hello
      Task 0 is in state 3
      . . . . . .

      The log on master says:

      I1010 14:48:44.420087 19698 master.cpp:7766] Sending 1 offers to framework 00afc79a-cb51-4f9d-8c78-6000253ae396-0000 (Docker No Executor Framework (C++)) at scheduler-2beb12b1-971d-4f05-b023-498665053465@127.0.1.1:45628
      W1010 14:48:44.422901 19701 master.cpp:3911] Implicitly declining offers: [ 00afc79a-cb51-4f9d-8c78-6000253ae396-O1 ] in ACCEPT call for framework 00afc79a-cb51-4f9d-8c78-6000253ae396-0000 as the launch operation specified no tasks
      I1010 14:48:44.423539 19701 master.cpp:9389] Removing offer 00afc79a-cb51-4f9d-8c78-6000253ae396-O1
      I1010 14:48:44.423607 19701 master.cpp:4196] Processing ACCEPT call for offers: [ 00afc79a-cb51-4f9d-8c78-6000253ae396-O1 ] on agent 00afc79a-cb51-4f9d-8c78-6000253ae396-S0 at slave(1)@127.0.1.1:5051 (pas-lab-server6) for framework 00afc79a-cb51-4f9d-8c78-6000253ae396-0000 (Docker No Executor Framework (C++)) at scheduler-2beb12b1-971d-4f05-b023-498665053465@127.0.1.1:45628
      I1010 14:48:44.425109 19691 hierarchical.cpp:887] Updated allocation of framework 00afc79a-cb51-4f9d-8c78-6000253ae396-0000 on agent 00afc79a-cb51-4f9d-8c78-6000253ae396-S0 from gpus(allocated: *):2; cpus(allocated: *):40; mem(allocated: *):63421; disk(allocated: *):868675; ports(allocated: *):[31000-32000] to gpus(allocated: *):2; cpus(allocated: *):40; mem(allocated: *):63421; disk(allocated: *):868675; ports(allocated: *):[31000-32000]
      
      

      Attachments

        Activity

          People

            Unassigned Unassigned
            ambujpandey.ap Ambuj Pandey
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: