[MESOS-9765] Test `ROOT_CreateDestroyPersistentMountVolumeWithReboot` is flaky. - ASF JIRA

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 1.9.0
Component/s: test
Labels:
- flaky-test

Sprint:
Storage: RI-13 Sprint 45, Storage: RI-14 Sprint 46
Story Points:
1

Description

Observed a failure on test CSIVersion/StorageLocalResourceProviderTest.ROOT_CreateDestroyPersistentMountVolumeWithReboot/v0 with the following error:

unknown file: Failure

Unexpected mock function call - returning directly.
    Function call: statusUpdate(0x7ffea80beb70, @0x7f6027ff6b90 136-byte object <E0-A2 02-36 60-7F 00-00 00-00 00-00 00-00 00-00 3E-EC 00-00 00-00 00-00 B8-0C B0-33 60-7F 00-00 40-7C 0A-04 60-7F 00-00 30-0A 0C-04 60-7F 00-00 F0-7D 07-04 60-7F 00-00 20-88 0B-04 60-7F 00-00 ... 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 A0-28 0C-04 60-7F 00-00 E7-52 88-A9 71-4C D7-41 00-00 00-00 01-00 00-00 08-00 00-00 03-00 00-00>)
Google Mock tried the following 6 expectations, but none matched:

/home/centos/workspace/mesos/Mesos_CI-build/FLAG/CMake/label/mesos-ec2-centos-7/mesos/src/tests/storage_local_resource_provider_tests.cpp:2552: tried expectation #0: EXPECT_CALL(sched, statusUpdate(&driver, TaskStatusStateEq(TASK_STARTING)))...
  Expected arg #1: task status state eq TASK_STARTING
           Actual: 136-byte object <E0-A2 02-36 60-7F 00-00 00-00 00-00 00-00 00-00 3E-EC 00-00 00-00 00-00 B8-0C B0-33 60-7F 00-00 40-7C 0A-04 60-7F 00-00 30-0A 0C-04 60-7F 00-00 F0-7D 07-04 60-7F 00-00 20-88 0B-04 60-7F 00-00 ... 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 A0-28 0C-04 60-7F 00-00 E7-52 88-A9 71-4C D7-41 00-00 00-00 01-00 00-00 08-00 00-00 03-00 00-00>
         Expected: to be called once
           Actual: called once - saturated and active
/home/centos/workspace/mesos/Mesos_CI-build/FLAG/CMake/label/mesos-ec2-centos-7/mesos/src/tests/storage_local_resource_provider_tests.cpp:2553: tried expectation #1: EXPECT_CALL(sched, statusUpdate(&driver, TaskStatusStateEq(TASK_RUNNING)))...
  Expected arg #1: task status state eq TASK_RUNNING
           Actual: 136-byte object <E0-A2 02-36 60-7F 00-00 00-00 00-00 00-00 00-00 3E-EC 00-00 00-00 00-00 B8-0C B0-33 60-7F 00-00 40-7C 0A-04 60-7F 00-00 30-0A 0C-04 60-7F 00-00 F0-7D 07-04 60-7F 00-00 20-88 0B-04 60-7F 00-00 ... 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 A0-28 0C-04 60-7F 00-00 E7-52 88-A9 71-4C D7-41 00-00 00-00 01-00 00-00 08-00 00-00 03-00 00-00>
         Expected: to be called once
           Actual: called once - saturated and active
/home/centos/workspace/mesos/Mesos_CI-build/FLAG/CMake/label/mesos-ec2-centos-7/mesos/src/tests/storage_local_resource_provider_tests.cpp:2554: tried expectation #2: EXPECT_CALL(sched, statusUpdate(&driver, TaskStatusStateEq(TASK_FINISHED)))...
  Expected arg #1: task status state eq TASK_FINISHED
           Actual: 136-byte object <E0-A2 02-36 60-7F 00-00 00-00 00-00 00-00 00-00 3E-EC 00-00 00-00 00-00 B8-0C B0-33 60-7F 00-00 40-7C 0A-04 60-7F 00-00 30-0A 0C-04 60-7F 00-00 F0-7D 07-04 60-7F 00-00 20-88 0B-04 60-7F 00-00 ... 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 A0-28 0C-04 60-7F 00-00 E7-52 88-A9 71-4C D7-41 00-00 00-00 01-00 00-00 08-00 00-00 03-00 00-00>
         Expected: to be called once
           Actual: called once - saturated and active
/home/centos/workspace/mesos/Mesos_CI-build/FLAG/CMake/label/mesos-ec2-centos-7/mesos/src/tests/storage_local_resource_provider_tests.cpp:2630: tried expectation #3: EXPECT_CALL(sched, statusUpdate(&driver, TaskStatusStateEq(TASK_STARTING)))...
  Expected arg #1: task status state eq TASK_STARTING
           Actual: 136-byte object <E0-A2 02-36 60-7F 00-00 00-00 00-00 00-00 00-00 3E-EC 00-00 00-00 00-00 B8-0C B0-33 60-7F 00-00 40-7C 0A-04 60-7F 00-00 30-0A 0C-04 60-7F 00-00 F0-7D 07-04 60-7F 00-00 20-88 0B-04 60-7F 00-00 ... 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 A0-28 0C-04 60-7F 00-00 E7-52 88-A9 71-4C D7-41 00-00 00-00 01-00 00-00 08-00 00-00 03-00 00-00>
         Expected: to be called once
           Actual: called once - saturated and active
/home/centos/workspace/mesos/Mesos_CI-build/FLAG/CMake/label/mesos-ec2-centos-7/mesos/src/tests/storage_local_resource_provider_tests.cpp:2631: tried expectation #4: EXPECT_CALL(sched, statusUpdate(&driver, TaskStatusStateEq(TASK_RUNNING)))...
  Expected arg #1: task status state eq TASK_RUNNING
           Actual: 136-byte object <E0-A2 02-36 60-7F 00-00 00-00 00-00 00-00 00-00 3E-EC 00-00 00-00 00-00 B8-0C B0-33 60-7F 00-00 40-7C 0A-04 60-7F 00-00 30-0A 0C-04 60-7F 00-00 F0-7D 07-04 60-7F 00-00 20-88 0B-04 60-7F 00-00 ... 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 A0-28 0C-04 60-7F 00-00 E7-52 88-A9 71-4C D7-41 00-00 00-00 01-00 00-00 08-00 00-00 03-00 00-00>
         Expected: to be called once
           Actual: never called - unsatisfied and active
/home/centos/workspace/mesos/Mesos_CI-build/FLAG/CMake/label/mesos-ec2-centos-7/mesos/src/tests/storage_local_resource_provider_tests.cpp:2632: tried expectation #5: EXPECT_CALL(sched, statusUpdate(&driver, TaskStatusStateEq(TASK_FINISHED)))...
  Expected arg #1: task status state eq TASK_FINISHED
           Actual: 136-byte object <E0-A2 02-36 60-7F 00-00 00-00 00-00 00-00 00-00 3E-EC 00-00 00-00 00-00 B8-0C B0-33 60-7F 00-00 40-7C 0A-04 60-7F 00-00 30-0A 0C-04 60-7F 00-00 F0-7D 07-04 60-7F 00-00 20-88 0B-04 60-7F 00-00 ... 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 A0-28 0C-04 60-7F 00-00 E7-52 88-A9 71-4C D7-41 00-00 00-00 01-00 00-00 08-00 00-00 03-00 00-00>
         Expected: to be called once
           Actual: never called - unsatisfied and active

The problem is that the task was OOM-killed:

I0503 01:04:24.410918 24105 memory.cpp:515] OOM detected for container df47d711-3ee8-4430-b90d-b29bcb90d407
I0503 01:04:24.411118 24105 memory.cpp:555] Memory limit exceeded: Requested: 32MB Maximum Used: 32MB

MEMORY STATISTICS:
cache 471040
rss 33083392
rss_huge 14680064
mapped_file 0
swap 0
pgpgin 4662
pgpgout 47
pgfault 77439
pgmajfault 0
inactive_anon 471040
active_anon 33083392
inactive_file 0
active_file 0
unevictable 0
hierarchical_memory_limit 33554432
hierarchical_memsw_limit 9223372036854771712
total_cache 471040
total_rss 33083392
total_rss_huge 14680064
total_mapped_file 0
total_swap 0
total_pgpgin 4662
total_pgpgout 47
total_pgfault 77439
total_pgmajfault 0
total_inactive_anon 471040
total_active_anon 33083392
total_inactive_file 0
total_active_file 0
total_unevictable 0
I0503 01:04:24.412195 24105 containerizer.cpp:3147] Container df47d711-3ee8-4430-b90d-b29bcb90d407 has reached its limit for resource [{"name":"mem","scalar":{"value":32.0},"type":"SCALAR"}] and will be terminated
I0503 01:04:24.412314 24105 containerizer.cpp:2586] Destroying container df47d711-3ee8-4430-b90d-b29bcb90d407 in RUNNING state

This might happen to other SLRP root tests as well.

Test `ROOT_CreateDestroyPersistentMountVolumeWithReboot` is flaky.

Details

Description

Attachments

Activity

People

Dates