Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-9196

Removing rootfs mounts may fail with EBUSY.

Attach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      We observed in production environment that this

      Failed to destroy the provisioned rootfs when destroying container: Collect failed: Failed to destroy overlay-mounted rootfs '/var/lib/mesos/slave/provisioner/containers/6332cf3d-9897-475b-88b3-40e983a2a531/containers/e8f36ad7-c9ae-40da-9d14-431e98174735/backends/overlay/rootfses/d601ef1b-11b9-445a-b607-7c6366cd21ec': Failed to unmount '/var/lib/mesos/slave/provisioner/containers/6332cf3d-9897-475b-88b3-40e983a2a531/containers/e8f36ad7-c9ae-40da-9d14-431e98174735/backends/overlay/rootfses/d601ef1b-11b9-445a-b607-7c6366cd21ec': Device or resource busy
      

      Consider fixing the issue by using detach unmount when unmounting container rootfs. See MESOS-3349 for details.

      The root cause on why "Device or resource busy" is received when doing rootfs unmount is still unknown.

      UPDATE: The production environment has a cronjob that scan filesystems to build index (updatedb for mlocate). This can explain the EBUSY we receive when doing `unmount`.

      UPDATE: Splunk that's scanning `/var/lib/mesos` could also be a source of triggers.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            jieyu Jie Yu
            gilbert Gilbert Song
            Gilbert Song Gilbert Song
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Agile

                Completed Sprint:
                Mesosphere Sprint 2018-28 ended 15/Sep/18
                View on Board

                Slack

                  Issue deployment