Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Not A Bug
-
Impala 2.12.0
-
None
-
ghx-label-2
Description
On a recent rhel exhaustive test (kicked off on 12/21):
verifiers/test_verify_metrics.py:34: in test_metrics_are_zero
verifier.verify_metrics_are_zero()
verifiers/metric_verifier.py:47: in verify_metrics_are_zero
self.wait_for_metric(metric, 0, timeout)
verifiers/metric_verifier.py:62: in wait_for_metric
self.impalad_service.wait_for_metric_value(metric_name, expected_value, timeout)
common/impala_service.py:89: in wait_for_metric_value
(metric_name, expected_value, timeout)
E AssertionError: Metric value impala-server.mem-pool.total-bytes did not reach value 0 in 60s
Here is the metric that was obtained:
...
MainThread: Getting metric: impala-server.mem-pool.total-bytes from localhost:25000
MainThread: Waiting for metric value 'impala-server.mem-pool.total-bytes'=0. Current value: 3290112
MainThread: Sleeping 1s before next retry.
MainThread: Getting metric: impala-server.mem-pool.total-bytes from localhost:25000
MainThread: Waiting for metric value 'impala-server.mem-pool.total-bytes'=0. Current value: 3290112
MainThread: Sleeping 1s before next retry.
Similarly, the number of unused buffers did not reach the expected value:
verifiers/test_verify_metrics.py:39: in test_num_unused_buffers
verifier.verify_num_unused_buffers()
verifiers/metric_verifier.py:59: in verify_num_unused_buffers
timeout)
verifiers/metric_verifier.py:62: in wait_for_metric
self.impalad_service.wait_for_metric_value(metric_name, expected_value, timeout)
common/impala_service.py:89: in wait_for_metric_value
(metric_name, expected_value, timeout)
E AssertionError: Metric value impala-server.io-mgr.num-unused-buffers did not reach value 204 in 60s
And the value that was observed:
....
MainThread: Getting metric: impala-server.io-mgr.num-unused-buffers from localhost:25000
MainThread: Waiting for metric value 'impala-server.io-mgr.num-unused-buffers'=204. Current value: 203
MainThread: Sleeping 1s before next retry.
MainThread: Getting metric: impala-server.io-mgr.num-unused-buffers from localhost:25000
MainThread: Waiting for metric value 'impala-server.io-mgr.num-unused-buffers'=204. Current value: 203
MainThread: Sleeping 1s before next retry.
MainThread: Getting metric: impala-server.io-mgr.num-unused-buffers from localhost:25000
MainThread: Waiting for metric value 'impala-server.io-mgr.num-unused-buffers'=204. Current value: 203
MainThread: Sleeping 1s before next retry.
Attachments
Issue Links
- is duplicated by
-
IMPALA-6443 Memory leak on exhaustive rhel7 test runs
- Resolved
-
IMPALA-6352 TestTableSample took too long in recent tests
- Closed
- is related to
-
IMPALA-6352 TestTableSample took too long in recent tests
- Closed