Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-9462

Devices in a container are inaccessible due to `nodev` on `/var/run`.

    XMLWordPrintableJSON

Details

    • Containerization R8 Sprint 35
    • 3

    Description

      A recent patch (commit ede8155d1d043137e15007c48da36ac5fa0b5124) changes the behavior of how standard device nodes (e.g., /dev/null, etc.) are setup. It uses bind mount (from host) now (instead of mknod).

      The devices nodes are created under `/var/run/mesos/containers/<container_id>/devices`, and then bind mounted to the container root filesystem. This is problematic for those Linux distros that mount `/var/run` (or `/run`) as `nodev`. For instance, CentOS 7.4:

      [jie@core-dev ~]$ cat /proc/self/mountinfo | grep "/run\ "                                                                                                                         
      24 62 0:19 / /run rw,nosuid,nodev shared:23 - tmpfs tmpfs rw,seclabel,mode=755
      [jie@core-dev ~]$ cat /etc/redhat-release 
      CentOS Linux release 7.4.1708 (Core) 
      

      As a result, the `/dev/null` devices in the container will inherit the `nodev` from `/run` on the host

      629 625 0:121 /mesos/containers/49f1da14-d741-4030-994c-0d8ed5093b13/devices/null /dev/null rw,nosuid,nodev - tmpfs tmpfs rw,mode=755
      

      This will cause "Permission Denied" error when a process in the container tries to open the device node.

      You can try to reproduce this issue using Mesos Mini

      docker run --rm --privileged -p 5050:5050 -p 5051:5051 -p 8080:8080 mesos/mesos-mini:master-2018-12-06
      

      And the, go to Marathon UI (http://localhost:8080), and launch an app using the following config

      {
        "id": "/test",
        "cmd": "dd if=/dev/zero of=file bs=1024 count=1 oflag=dsync",
        "cpus": 1,
        "mem": 128,
        "disk": 128,
        "instances": 1,
        "container": {
          "type": "MESOS",
          "docker": {
            "image": "ubuntu:18.04"
          }
        }
      }
      

      You'll see the task failed with "Permission Denied".

      The task will run normally if you use `mesos/mesos-mini:master-2018-12-01`

      Attachments

        Issue Links

          Activity

            People

              abudnik Andrei Budnik
              jieyu Jie Yu
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: