Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
1.5.1, 1.6.1, 1.7.0
Description
When we authorize some actions (right now I see this happening to create / destroy volumes, reserve / unreserve resources) and authorizer fails (i.e. returns the future in non-ready state), an assertion is triggered:
mesos-master[49173]: F1015 11:40:29.795748 49396 future.hpp:1306] Check failed: !isFailed() Future::get() but state == FAILED: Failed to retrieve permissions from IAM at url https://localhost:443/acs/api/v1/users/marathon_user_ee/permissions the request failed: Failed to contact bouncer at https://localhost:443/acs/api/v1/users/marathon_user_ee/permissions due to time out after 3 attempts
This is due to incorrect assumption in our code, see for example https://github.com/apache/mesos/blob/a063afce9868dcee38a0ab7efaa028244f3999cf/src/master/master.cpp#L3752-L3763:
return await(authorizations)
.then([](const vector<Future<bool>>& authorizations)
-> Future<bool> {
// Compute a disjunction.
foreach (const Future<bool>& authorization, authorizations) {
if (!authorization.get()) {
return false;
}
}
return true;
});
Futures returned from await are guaranteed to be in terminal state, but not necessarily ready! In the snippet above, !authorization.get() is invoked without being checked ⇒ assertion fails.
Full stack trace:
Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: F1015 11:40:29.795748 49396 future.hpp:1306] Check failed: !isFailed() Future::get() but state == FAILED: Failed to retrieve permissions from IAM at url https://localhost:443/acs/api/v1/users/marathon_user_ee/permissions the request failed: Failed to contact bouncer at https://localhost:443/acs/api/v1/users/marathon_user_ee/permissions due to time out after 3 attemptsF1015 11:40:29.796037 49395 future.hpp:1306] Check failed: !isFailed() Future::get() but state == FAILED: Failed to retrieve permissions from IAM at url https://localhost:443/acs/api/v1/users/marathon_user_ee/permissions the request failed: Failed to contact bouncer at https://localhost:443/acs/api/v1/users/marathon_user_ee/permissions due to time out after 3 attemptsF1015 11:40:29.796097 49384 future.hpp:1306] Check failed: !isFailed() Future::get() but state == FAILED: Failed to retrieve permissions from IAM at url https://localhost:443/acs/api/v1/users/marathon_user_ee/permissions the request failed: Failed to contact bouncer at https://localhost:443/acs/api/v1/users/marathon_user_ee/permissions due to time out after 3 attemptsF1015 11:40:29.796249 49393 future.hpp:1306] Check failed: !isFailed() Future::get() but state == FAILED: Failed to retrieve permissions from IAM at url https://localhost:443/acs/api/v1/users/marathon_user_ee/permissions the request failed: Failed to contact bouncer at https://localhost:443/acs/api/v1/users/marathon_user_ee/permissions due to time out after 3 attemptsF1015 11:40:29.796375 49390 future.hpp:1306] Check failed: !isFailed() Future::get() but state == FAILED: Failed to retrieve permissions from IAM at url https://localhost:443/acs/api/v1/users/marathon_user_ee/permissions the request failed: Failed to contact bouncer at https://localhost:443/acs/api/v1/users/marathon_user_ee/permissions due to time out after 3 attemptsF1015 11:40:29.796483 49388 future.hpp:1306] Check failed: !isFailed() Future::get() but state == FAILED: Failed to retrieve permissions from IAM at url https://localhost:443/acs/api/v1/users/marathon_user_ee/permissions the request failed: Failed to contact bouncer at https://localhost:443/acs/api/v1/users/marathon_user_ee/permissions due to time out after 3 attemptsF1015 11:40:29.796629 49381 future.hpp:1306] Check failed: !isFailed() Future::get() but state == FAILED: Failed to retrieve permissions from IAM at url https://localhost:443/acs/api/v1/users/marathon_user_ee/permissions the request failed: Failed to contact bouncer at https://localhost:443/acs/api/v1/users/marathon_user_ee/permissions due to time out after 3 attemptsF1015 11:40:29.796700 49385 future.hpp:1306] Check failed: !isFailed() Future::get() but state == FAILED: Failed to retrieve permissions from IAM at url https://localhost:443/acs/api/v1/users/marathon_user_ee/permissions the request failed: Failed to contact bouncer at https://localhost:443/acs/api/v1/users/marathon_user_ee/permissions due to time out after 3 attemptsF1015 11:40:29.796780 49386 future.hpp:1306] Check failed: !isFailed() Future::get() but state == FAILED: Failed to retrieve permissions from IAM at url https://localhost:443/acs/api/v1/users/marathon_user_ee/permissions the request failed: Failed to contact bouncer at https://localhost:443/acs/api/v1/users/marathon_user_ee/permissions due to time out after 3 attemptsF1015 11:40:29.796893 49391 future.hpp:1306] Check failed: !isFailed() Future::get() but state == FAILED: Failed to retrieve permissions from IAM at url https://localhost:443/acs/api/v1/users/marathon_user_ee/permissions the request failed: Failed to contact bouncer at https://localhost:443/acs/api/v1/users/marathon_user_ee/permissions due to time out after 3 attemptsF1015 11:40:29.797003 49392 future.hpp:1306] Check failed: !isFailed() Future::get() but state == FAILED: Failed to retrieve permissions from IAM at url https://localhost:443/acs/api/v1/users/marathon_user_ee/permissions the request failed: Failed to contact bouncer at https://localhost:443/acs/api/v1/users/marathon_user_ee/permissions due to time out after 3 attemptsF1015 11:40:29.797868 49394 future.hpp:1306] Check failed: !isFailed() Future::get() but state == FAILED: Failed to retrieve permissions from IAM at url https://localhost:443/acs/api/v1/users/marathon_user_ee/permissions the request failed: Failed to contact bouncer at https://localhost:443/acs/api/v1/users/marathon_user_ee/permissions due to time out after 3 attemptsF1015 11:40:29.798266 49389 future.hpp:1306] Check failed: !isFailed() Future::get() but state == FAILED: Failed to retrieve permissions from IAM at url https://localhost:443/acs/api/v1/users/marathon_user_ee/permissions the request failed: Failed to contact bouncer at https://localhost:443/acs/api/v1/users/marathon_user_ee/permissions due to time out after 3 attemptsF1015 11:40:29.799608 49387 future.hpp:1306] Check failed: !isFailed() Future::get() but state == FAILED: Failed to retrieve permissions from IAM at url https://localhost:443/acs/api/v1/users/marathon_user_ee/permissions the request failed: Failed to contact bouncer at https://localhost:443/acs/api/v1/users/marathon_user_ee/permissions due to time out after 3 attempts Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: *** Check failure stack trace: *** Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e7199fd google::LogMessage::Fail() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e7199fd google::LogMessage::Fail() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e7199fd google::LogMessage::Fail() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e7199fd google::LogMessage::Fail() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e7199fd google::LogMessage::Fail() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e7199fd google::LogMessage::Fail() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e7199fd google::LogMessage::Fail() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e7199fd google::LogMessage::Fail() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e7199fd google::LogMessage::Fail() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e7199fd google::LogMessage::Fail() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e7199fd google::LogMessage::Fail() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e7199fd google::LogMessage::Fail() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e71b82d google::LogMessage::SendToLog() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e71b82d google::LogMessage::SendToLog() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e71b82d google::LogMessage::SendToLog() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e7199fd google::LogMessage::Fail() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e7199fd google::LogMessage::Fail() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e7199fd google::LogMessage::Fail() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e71b82d google::LogMessage::SendToLog() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e71b82d google::LogMessage::SendToLog() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e71b82d google::LogMessage::SendToLog() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e71b82d google::LogMessage::SendToLog() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e71b82d google::LogMessage::SendToLog() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e71b82d google::LogMessage::SendToLog() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e71b82d google::LogMessage::SendToLog() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e71b82d google::LogMessage::SendToLog() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e71b82d google::LogMessage::SendToLog() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e71b82d google::LogMessage::SendToLog() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e71b82d google::LogMessage::SendToLog() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e71b82d google::LogMessage::SendToLog() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e7195ec google::LogMessage::Flush() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e7195ec google::LogMessage::Flush() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e7195ec google::LogMessage::Flush() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e7195ec google::LogMessage::Flush() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e7195ec google::LogMessage::Flush() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e7195ec google::LogMessage::Flush() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e7195ec google::LogMessage::Flush() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e7195ec google::LogMessage::Flush() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e7195ec google::LogMessage::Flush() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e7195ec google::LogMessage::Flush() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e7195ec google::LogMessage::Flush() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e7195ec google::LogMessage::Flush() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e71c129 google::LogMessageFatal::~LogMessageFatal() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e71c129 google::LogMessageFatal::~LogMessageFatal() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e71c129 google::LogMessageFatal::~LogMessageFatal() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e7195ec google::LogMessage::Flush() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e7195ec google::LogMessage::Flush() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e7195ec google::LogMessage::Flush() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e71c129 google::LogMessageFatal::~LogMessageFatal() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e71c129 google::LogMessageFatal::~LogMessageFatal() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e71c129 google::LogMessageFatal::~LogMessageFatal() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e71c129 google::LogMessageFatal::~LogMessageFatal() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e71c129 google::LogMessageFatal::~LogMessageFatal() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e71c129 google::LogMessageFatal::~LogMessageFatal() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e71c129 google::LogMessageFatal::~LogMessageFatal() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e71c129 google::LogMessageFatal::~LogMessageFatal() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d4f97e3 process::Future<>::get() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d4f97e3 process::Future<>::get() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d4f97e3 process::Future<>::get() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d7071b1 _ZNO6lambda12CallableOnceIFN7process6FutureIbEERKSt6vectorIS3_SaIS3_EEEE10CallableFnIZN5mesos8internal6master6Master21authorizeCreateVolumeERKNSC_22Offer_Operation_CreateERK6OptionINS1_4http14authentication9PrincipalEEEUlS8_E_EclES8_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d7070b1 _ZNO6lambda12CallableOnceIFN7process6FutureIbEERKSt6vectorIS3_SaIS3_EEEE10CallableFnIZN5mesos8internal6master6Master25authorizeReserveResourcesERKNSC_9ResourcesERK6OptionINS1_4http14authentication9PrincipalEEEUlS8_E_EclES8_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e71c129 google::LogMessageFatal::~LogMessageFatal() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d7070b1 _ZNO6lambda12CallableOnceIFN7process6FutureIbEERKSt6vectorIS3_SaIS3_EEEE10CallableFnIZN5mesos8internal6master6Master25authorizeReserveResourcesERKNSC_9ResourcesERK6OptionINS1_4http14authentication9PrincipalEEEUlS8_E_EclES8_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d790c91 process::internal::thenf<>() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d790c91 process::internal::thenf<>() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d790c91 process::internal::thenf<>() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e71c129 google::LogMessageFatal::~LogMessageFatal() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e71c129 google::LogMessageFatal::~LogMessageFatal() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d4f97e3 process::Future<>::get() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e71c129 google::LogMessageFatal::~LogMessageFatal() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d4f97e3 process::Future<>::get() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d4f97e3 process::Future<>::get() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d4f97e3 process::Future<>::get() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d7070b1 _ZNO6lambda12CallableOnceIFN7process6FutureIbEERKSt6vectorIS3_SaIS3_EEEE10CallableFnIZN5mesos8internal6master6Master25authorizeReserveResourcesERKNSC_9ResourcesERK6OptionINS1_4http14authentication9PrincipalEEEUlS8_E_EclES8_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d4f97e3 process::Future<>::get() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d7071b1 _ZNO6lambda12CallableOnceIFN7process6FutureIbEERKSt6vectorIS3_SaIS3_EEEE10CallableFnIZN5mesos8internal6master6Master21authorizeCreateVolumeERKNSC_22Offer_Operation_CreateERK6OptionINS1_4http14authentication9PrincipalEEEUlS8_E_EclES8_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d4f97e3 process::Future<>::get() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d7070b1 _ZNO6lambda12CallableOnceIFN7process6FutureIbEERKSt6vectorIS3_SaIS3_EEEE10CallableFnIZN5mesos8internal6master6Master25authorizeReserveResourcesERKNSC_9ResourcesERK6OptionINS1_4http14authentication9PrincipalEEEUlS8_E_EclES8_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d7071b1 _ZNO6lambda12CallableOnceIFN7process6FutureIbEERKSt6vectorIS3_SaIS3_EEEE10CallableFnIZN5mesos8internal6master6Master21authorizeCreateVolumeERKNSC_22Offer_Operation_CreateERK6OptionINS1_4http14authentication9PrincipalEEEUlS8_E_EclES8_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d765405 _ZNO6lambda12CallableOnceIFvRKN7process6FutureISt6vectorINS2_IbEESaIS4_EEEEEE10CallableFnINS_8internal7PartialIPFvONS0_IFS4_RKS6_EEESt10unique_ptrINS1_7PromiseIbEESt14default_deleteISM_EES9_EJSI_SP_St12_PlaceholderILi1EEEEEEclES9_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d4f97e3 process::Future<>::get() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d7070b1 _ZNO6lambda12CallableOnceIFN7process6FutureIbEERKSt6vectorIS3_SaIS3_EEEE10CallableFnIZN5mesos8internal6master6Master25authorizeReserveResourcesERKNSC_9ResourcesERK6OptionINS1_4http14authentication9PrincipalEEEUlS8_E_EclES8_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d7071b1 _ZNO6lambda12CallableOnceIFN7process6FutureIbEERKSt6vectorIS3_SaIS3_EEEE10CallableFnIZN5mesos8internal6master6Master21authorizeCreateVolumeERKNSC_22Offer_Operation_CreateERK6OptionINS1_4http14authentication9PrincipalEEEUlS8_E_EclES8_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d4f97e3 process::Future<>::get() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d765405 _ZNO6lambda12CallableOnceIFvRKN7process6FutureISt6vectorINS2_IbEESaIS4_EEEEEE10CallableFnINS_8internal7PartialIPFvONS0_IFS4_RKS6_EEESt10unique_ptrINS1_7PromiseIbEESt14default_deleteISM_EES9_EJSI_SP_St12_PlaceholderILi1EEEEEEclES9_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d765405 _ZNO6lambda12CallableOnceIFvRKN7process6FutureISt6vectorINS2_IbEESaIS4_EEEEEE10CallableFnINS_8internal7PartialIPFvONS0_IFS4_RKS6_EEESt10unique_ptrINS1_7PromiseIbEESt14default_deleteISM_EES9_EJSI_SP_St12_PlaceholderILi1EEEEEEclES9_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d7071b1 _ZNO6lambda12CallableOnceIFN7process6FutureIbEERKSt6vectorIS3_SaIS3_EEEE10CallableFnIZN5mesos8internal6master6Master21authorizeCreateVolumeERKNSC_22Offer_Operation_CreateERK6OptionINS1_4http14authentication9PrincipalEEEUlS8_E_EclES8_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d7071b1 _ZNO6lambda12CallableOnceIFN7process6FutureIbEERKSt6vectorIS3_SaIS3_EEEE10CallableFnIZN5mesos8internal6master6Master21authorizeCreateVolumeERKNSC_22Offer_Operation_CreateERK6OptionINS1_4http14authentication9PrincipalEEEUlS8_E_EclES8_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d4f97e3 process::Future<>::get() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d7070b1 _ZNO6lambda12CallableOnceIFN7process6FutureIbEERKSt6vectorIS3_SaIS3_EEEE10CallableFnIZN5mesos8internal6master6Master25authorizeReserveResourcesERKNSC_9ResourcesERK6OptionINS1_4http14authentication9PrincipalEEEUlS8_E_EclES8_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d788169 _ZN7process8internal3runIN6lambda12CallableOnceIFvRKNS_6FutureISt6vectorINS4_IbEESaIS6_EEEEEEEJRS9_EEEvOS5_IT_SaISF_EEDpOT0_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d788169 _ZN7process8internal3runIN6lambda12CallableOnceIFvRKNS_6FutureISt6vectorINS4_IbEESaIS6_EEEEEEEJRS9_EEEvOS5_IT_SaISF_EEDpOT0_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d788169 _ZN7process8internal3runIN6lambda12CallableOnceIFvRKNS_6FutureISt6vectorINS4_IbEESaIS6_EEEEEEEJRS9_EEEvOS5_IT_SaISF_EEDpOT0_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d790c91 process::internal::thenf<>() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d790c91 process::internal::thenf<>() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d790c91 process::internal::thenf<>() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d790c91 process::internal::thenf<>() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d790c91 process::internal::thenf<>() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d790c91 process::internal::thenf<>() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d790c91 process::internal::thenf<>() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d790c91 process::internal::thenf<>() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d798154 process::Future<>::_set<>() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d790c91 process::internal::thenf<>() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d4f97e3 process::Future<>::get() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d798154 process::Future<>::_set<>() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d798154 process::Future<>::_set<>() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d7070b1 _ZNO6lambda12CallableOnceIFN7process6FutureIbEERKSt6vectorIS3_SaIS3_EEEE10CallableFnIZN5mesos8internal6master6Master25authorizeReserveResourcesERKNSC_9ResourcesERK6OptionINS1_4http14authentication9PrincipalEEEUlS8_E_EclES8_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d4f97e3 process::Future<>::get() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d765405 _ZNO6lambda12CallableOnceIFvRKN7process6FutureISt6vectorINS2_IbEESaIS4_EEEEEE10CallableFnINS_8internal7PartialIPFvONS0_IFS4_RKS6_EEESt10unique_ptrINS1_7PromiseIbEESt14default_deleteISM_EES9_EJSI_SP_St12_PlaceholderILi1EEEEEEclES9_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d7070b1 _ZNO6lambda12CallableOnceIFN7process6FutureIbEERKSt6vectorIS3_SaIS3_EEEE10CallableFnIZN5mesos8internal6master6Master25authorizeReserveResourcesERKNSC_9ResourcesERK6OptionINS1_4http14authentication9PrincipalEEEUlS8_E_EclES8_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d765405 _ZNO6lambda12CallableOnceIFvRKN7process6FutureISt6vectorINS2_IbEESaIS4_EEEEEE10CallableFnINS_8internal7PartialIPFvONS0_IFS4_RKS6_EEESt10unique_ptrINS1_7PromiseIbEESt14default_deleteISM_EES9_EJSI_SP_St12_PlaceholderILi1EEEEEEclES9_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d765405 _ZNO6lambda12CallableOnceIFvRKN7process6FutureISt6vectorINS2_IbEESaIS4_EEEEEE10CallableFnINS_8internal7PartialIPFvONS0_IFS4_RKS6_EEESt10unique_ptrINS1_7PromiseIbEESt14default_deleteISM_EES9_EJSI_SP_St12_PlaceholderILi1EEEEEEclES9_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d765405 _ZNO6lambda12CallableOnceIFvRKN7process6FutureISt6vectorINS2_IbEESaIS4_EEEEEE10CallableFnINS_8internal7PartialIPFvONS0_IFS4_RKS6_EEESt10unique_ptrINS1_7PromiseIbEESt14default_deleteISM_EES9_EJSI_SP_St12_PlaceholderILi1EEEEEEclES9_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d765405 _ZNO6lambda12CallableOnceIFvRKN7process6FutureISt6vectorINS2_IbEESaIS4_EEEEEE10CallableFnINS_8internal7PartialIPFvONS0_IFS4_RKS6_EEESt10unique_ptrINS1_7PromiseIbEESt14default_deleteISM_EES9_EJSI_SP_St12_PlaceholderILi1EEEEEEclES9_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d4f97e3 process::Future<>::get() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d765405 _ZNO6lambda12CallableOnceIFvRKN7process6FutureISt6vectorINS2_IbEESaIS4_EEEEEE10CallableFnINS_8internal7PartialIPFvONS0_IFS4_RKS6_EEESt10unique_ptrINS1_7PromiseIbEESt14default_deleteISM_EES9_EJSI_SP_St12_PlaceholderILi1EEEEEEclES9_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d765405 _ZNO6lambda12CallableOnceIFvRKN7process6FutureISt6vectorINS2_IbEESaIS4_EEEEEE10CallableFnINS_8internal7PartialIPFvONS0_IFS4_RKS6_EEESt10unique_ptrINS1_7PromiseIbEESt14default_deleteISM_EES9_EJSI_SP_St12_PlaceholderILi1EEEEEEclES9_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d7071b1 _ZNO6lambda12CallableOnceIFN7process6FutureIbEERKSt6vectorIS3_SaIS3_EEEE10CallableFnIZN5mesos8internal6master6Master21authorizeCreateVolumeERKNSC_22Offer_Operation_CreateERK6OptionINS1_4http14authentication9PrincipalEEEUlS8_E_EclES8_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d765405 _ZNO6lambda12CallableOnceIFvRKN7process6FutureISt6vectorINS2_IbEESaIS4_EEEEEE10CallableFnINS_8internal7PartialIPFvONS0_IFS4_RKS6_EEESt10unique_ptrINS1_7PromiseIbEESt14default_deleteISM_EES9_EJSI_SP_St12_PlaceholderILi1EEEEEEclES9_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d798355 process::internal::AwaitProcess<>::waited() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d765405 _ZNO6lambda12CallableOnceIFvRKN7process6FutureISt6vectorINS2_IbEESaIS4_EEEEEE10CallableFnINS_8internal7PartialIPFvONS0_IFS4_RKS6_EEESt10unique_ptrINS1_7PromiseIbEESt14default_deleteISM_EES9_EJSI_SP_St12_PlaceholderILi1EEEEEEclES9_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d798355 process::internal::AwaitProcess<>::waited() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d798355 process::internal::AwaitProcess<>::waited() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d788169 _ZN7process8internal3runIN6lambda12CallableOnceIFvRKNS_6FutureISt6vectorINS4_IbEESaIS6_EEEEEEEJRS9_EEEvOS5_IT_SaISF_EEDpOT0_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d788169 _ZN7process8internal3runIN6lambda12CallableOnceIFvRKNS_6FutureISt6vectorINS4_IbEESaIS6_EEEEEEEJRS9_EEEvOS5_IT_SaISF_EEDpOT0_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d788169 _ZN7process8internal3runIN6lambda12CallableOnceIFvRKNS_6FutureISt6vectorINS4_IbEESaIS6_EEEEEEEJRS9_EEEvOS5_IT_SaISF_EEDpOT0_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d788169 _ZN7process8internal3runIN6lambda12CallableOnceIFvRKNS_6FutureISt6vectorINS4_IbEESaIS6_EEEEEEEJRS9_EEEvOS5_IT_SaISF_EEDpOT0_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d788169 _ZN7process8internal3runIN6lambda12CallableOnceIFvRKNS_6FutureISt6vectorINS4_IbEESaIS6_EEEEEEEJRS9_EEEvOS5_IT_SaISF_EEDpOT0_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d788169 _ZN7process8internal3runIN6lambda12CallableOnceIFvRKNS_6FutureISt6vectorINS4_IbEESaIS6_EEEEEEEJRS9_EEEvOS5_IT_SaISF_EEDpOT0_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d788169 _ZN7process8internal3runIN6lambda12CallableOnceIFvRKNS_6FutureISt6vectorINS4_IbEESaIS6_EEEEEEEJRS9_EEEvOS5_IT_SaISF_EEDpOT0_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d788169 _ZN7process8internal3runIN6lambda12CallableOnceIFvRKNS_6FutureISt6vectorINS4_IbEESaIS6_EEEEEEEJRS9_EEEvOS5_IT_SaISF_EEDpOT0_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d790c91 process::internal::thenf<>() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d788169 _ZN7process8internal3runIN6lambda12CallableOnceIFvRKNS_6FutureISt6vectorINS4_IbEESaIS6_EEEEEEEJRS9_EEEvOS5_IT_SaISF_EEDpOT0_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d790c91 process::internal::thenf<>() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d790c91 process::internal::thenf<>() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d798154 process::Future<>::_set<>() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e669c51 process::ProcessBase::consume() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d798154 process::Future<>::_set<>() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d798154 process::Future<>::_set<>() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e669c51 process::ProcessBase::consume() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d798154 process::Future<>::_set<>() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d798154 process::Future<>::_set<>() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d798154 process::Future<>::_set<>() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e669c51 process::ProcessBase::consume() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d798154 process::Future<>::_set<>() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d798154 process::Future<>::_set<>() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d765405 _ZNO6lambda12CallableOnceIFvRKN7process6FutureISt6vectorINS2_IbEESaIS4_EEEEEE10CallableFnINS_8internal7PartialIPFvONS0_IFS4_RKS6_EEESt10unique_ptrINS1_7PromiseIbEESt14default_deleteISM_EES9_EJSI_SP_St12_PlaceholderILi1EEEEEEclES9_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d798154 process::Future<>::_set<>() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d765405 _ZNO6lambda12CallableOnceIFvRKN7process6FutureISt6vectorINS2_IbEESaIS4_EEEEEE10CallableFnINS_8internal7PartialIPFvONS0_IFS4_RKS6_EEESt10unique_ptrINS1_7PromiseIbEESt14default_deleteISM_EES9_EJSI_SP_St12_PlaceholderILi1EEEEEEclES9_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d765405 _ZNO6lambda12CallableOnceIFvRKN7process6FutureISt6vectorINS2_IbEESaIS4_EEEEEE10CallableFnINS_8internal7PartialIPFvONS0_IFS4_RKS6_EEESt10unique_ptrINS1_7PromiseIbEESt14default_deleteISM_EES9_EJSI_SP_St12_PlaceholderILi1EEEEEEclES9_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d798355 process::internal::AwaitProcess<>::waited() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d798355 process::internal::AwaitProcess<>::waited() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d798355 process::internal::AwaitProcess<>::waited() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d798355 process::internal::AwaitProcess<>::waited() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d798355 process::internal::AwaitProcess<>::waited() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d788169 _ZN7process8internal3runIN6lambda12CallableOnceIFvRKNS_6FutureISt6vectorINS4_IbEESaIS6_EEEEEEEJRS9_EEEvOS5_IT_SaISF_EEDpOT0_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d798355 process::internal::AwaitProcess<>::waited() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e6806cc process::ProcessManager::resume() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d798355 process::internal::AwaitProcess<>::waited() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d798355 process::internal::AwaitProcess<>::waited() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e6806cc process::ProcessManager::resume() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e6806cc process::ProcessManager::resume() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d788169 _ZN7process8internal3runIN6lambda12CallableOnceIFvRKNS_6FutureISt6vectorINS4_IbEESaIS6_EEEEEEEJRS9_EEEvOS5_IT_SaISF_EEDpOT0_ Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e686186 _ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vEEE6_M_runEv Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1d798355 process::internal::AwaitProcess<>::waited() Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1b353070 (unknown) Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1ab71e25 start_thread Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1e686186 _ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vEEE6_M_runEv Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1a89bbad __clone Oct 15 11:40:39 int-master2-mwst9.scaletesting.mesosphe.re mesos-master[49173]: @ 0x7fea1b353070 (unknown)