Description
With KUDU-3498 implemented in 8683b8bdb, now there are resource leak reports, see below.
Overall, the way how HashedWheelTimer is used for keeping scanners alive is in direct contradiction with the recommendation at this documentation page:
Do not create many instances.
HashedWheelTimer creates a new thread whenever it is instantiated and started. Therefore, you should make sure to create only one instance and share it across your application. One of the common mistakes, that makes your application unresponsive, is to create a new instance for every connection.
Probably, a better way of implementing the keep-alive feature for scanner objects in Kudu Java client would be reusing the HashedWheelTimer instance from corresponding AsyncKuduClient client instance, not creating a new instance of the timer (along with corresponding thread) per AsyncKuduScanner object. At least, an instance of HashedWheelTimer should be properly released/shutdown to avoid resource leakages (a running thread?) when GC-ing AsyncKuduScanner objects.
For example, below is an example how the leak is reported when running TestKuduClient.testStrings:
23:04:57.774 [ERROR - main] (ResourceLeakDetector.java:327) LEAK: HashedWheelTimer.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information. Recent access records: Created at: io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:312) io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:251) io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:224) io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:203) io.netty.util.HashedWheelTimer.<init>(HashedWheelTimer.java:185) org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:296) org.apache.kudu.client.AsyncKuduScanner.<init>(AsyncKuduScanner.java:431) org.apache.kudu.client.KuduScanner$KuduScannerBuilder.build(KuduScanner.java:260) org.apache.kudu.client.TestKuduClient.testStrings(TestKuduClient.java:692) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:748)
Attachments
Issue Links
- is caused by
-
KUDU-3498 Scanner keeps alive in periodically
-
- Resolved
-