Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-7883

Quota heuristic check not accounting for mount volumes

    XMLWordPrintableJSON

Details

    • Resource Mgmt RI12 Sp 42
    • 1

    Description

      This may be expected but came as a surprise to us. We are unable to create a quota bigger than the root disk space on slaves.

      Given two clusters with the same number of slaves and root disk size, but one that also has mount volumes, is what the disk resources look like:

      [root@fin-fang-foom-master-1 ~]# curl -s master.mesos:5050/state | jq '.slaves[] .resources .disk'
28698
28699
28698
28698
28697

      [root@hydra-master-1 ~]# curl -s master.mesos:5050/state | jq '.slaves[] .resources .disk'
50817
50817
50814
50819
50817

      In fin-fang-foom, I was able to create a quota for 143490mb which is the total of available disk resources, root in this case, as reported by Mesos. For hydra, I am only able to create a quota for 143489mb. This is equivalent to the total of root disks available in hydra rather than the total available disks reported by Mesos resources which is 254084mb.

      With a modified Mesos that adds logging to quota_handler, we can see that only the disk number increases in nonStaticClusterResources after every iteration. The final iteration is disk:143489 which is the maximum quota I was able to create on hydra. We expected that quota heuristic check would also include resources such as disk[MOUNT:/dcos/volume2]:7373

      Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.763764 24902 quota_handler.cpp:71] Performing capacity heuristic check for a set quota request
Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.763783 24902 quota_handler.cpp:87] heuristic: total quota 'disk(*):143489'

Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.763870 24902 quota_handler.cpp:111] heuristic: nonStaticAgentResources = 'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000]; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; disk(*):28698; cpus(*):4; mem(*):15023'
Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.763923 24902 quota_handler.cpp:113] heuristic: nonStaticClusterResources = 'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000]; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; disk(*):28698; cpus(*):4; mem(*):15023'


Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.763989 24902 quota_handler.cpp:111] heuristic: nonStaticAgentResources = 'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000]; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; disk(*):28698; cpus(*):4; mem(*):15023'
Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.764022 24902 quota_handler.cpp:113] heuristic: nonStaticClusterResources = 'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000]; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; disk(*):57396; cpus(*):8; mem(*):30046; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373'

Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.764077 24902 quota_handler.cpp:111] heuristic: nonStaticAgentResources = 'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000]; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; disk(*):28695; cpus(*):4; mem(*):15023'
Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.764119 24902 quota_handler.cpp:113] heuristic: nonStaticClusterResources = 'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000]; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; disk(*):86091; cpus(*):12; mem(*):45069; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373'

Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.764225 24902 quota_handler.cpp:111] heuristic: nonStaticAgentResources = 'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000]; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; disk(*):28700; cpus(*):4; mem(*):15023'
Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.764307 24902 quota_handler.cpp:113] heuristic: nonStaticClusterResources = 'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000]; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; disk(*):114791; cpus(*):16; mem(*):60092; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373'

Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.764434 24902 quota_handler.cpp:111] heuristic: nonStaticAgentResources = 'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000]; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; disk(*):28698; cpus(*):4; mem(*):15023'
Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.764492 24902 quota_handler.cpp:113] heuristic: nonStaticClusterResources = 'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000]; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; disk(*):143489; cpus(*):20; mem(*):75115; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373'

Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.764562 24902 quota_handler.cpp:118] heuristic: nonStaticClusterResources.contains(totalQuota)

      Attachments

        Issue Links

          relates to

          Task - A task that needs to be done. MESOS-6840 Tests for quota capacity heuristic.

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              bmahler Benjamin Mahler
              vroy Vincent Roy
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: