Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
0.12.0
-
None
-
None
-
- thrift latest master
- Operating Systems and Compilers:
- VS2017 x64
- VS2019 x64
- macOS 10.13
- Ubuntu 18.04 x86_64
- OpenSSL 1.1.1c (current latest official)
Description
We observe a memory corruption in SecurityTest. The issue is not fully reproducible: it appears on average in 1 out of 10 executions. However it is not dependent on the environment because can reproduce the problem on Windows VS2017 x64, VS2019 x64, macOS 10.13, and Ubuntu 18.04 x86_64.
On Linux the issue is often reported as:
[...] TEST: Server = TLSv1_2, Client = TLSv1_1 CLI 7f1be2eaa700 Exception: SSL_connect: tlsv1 alert protocol version (SSL_error_code = 1) Thrift: Mon Sep 2 07:51:32 2019 SSL_shutdown: shutdown while in init (SSL_error_code = 1) SRV 7f1be38bd700 Exception: SSL_accept: error code: 0 (SSL_error_code = 5) error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version Thrift: Mon Sep 2 07:51:32 2019 SSL_shutdown: shutdown while in init (SSL_error_code = 1) double free or corruption (out) unknown location(0): fatal error: in "SecurityTest/ssl_security_matrix": signal: SIGABRT (application abort requested) /builds/thrift/lib/cpp/test/SecurityTest.cpp(173): last checkpoint
But other forms also appear, for example:
[...]
Thrift: Mon Sep 2 07:50:53 2019 SSL_shutdown: shutdown while in init (SSL_error_code = 1)
TEST: Server = TLSv1_2, Client = TLSv1_2
corrupted size vs. prev_size
We tried to isolate a call stack for the problem but have failed so far. The boost message log does not always point to the same protocol combination. We executed the test in `valgrind` but it does never crash there. With `gdb` we can create a stack trace but it does not mean much to me:
EST: Server = TLSv1_2, Client = TLSv1_0 [New Thread 0x7f940fd05700 (LWP 1903)] [New Thread 0x7f9410718700 (LWP 1904)] CLI 7f9410718700 Exception: SSL_connect: tlsv1 alert protocol version (SSL_error_code = 1) Thrift: Mon Sep 2 08:36:14 2019 SSL_shutdown: shutdown while in init (SSL_error_code = 1) SRV 7f940fd05700 Exception: SSL_accept: error code: 0 (SSL_error_code = 5) error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version Thrift: Mon Sep 2 08:36:14 2019 SSL_shutdown: shutdown while in init (SSL_error_code = 1) double free or corruption (out) [Thread 0x7f9410718700 (LWP 1904) exited] Thread 28 "SecurityTest" received signal SIGABRT, Aborted. [Switching to Thread 0x7f940fd05700 (LWP 1903)] __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. (gdb) bt #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 #1 0x00007f9410b73801 in __GI_abort () at abort.c:79 #2 0x00007f9410bbc897 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7f9410ce9b9a "%s\n") at ../sysdeps/posix/libc_fatal.c:181 #3 0x00007f9410bc390a in malloc_printerr (str=str@entry=0x7f9410ceb870 "double free or corruption (out)") at malloc.c:5350 #4 0x00007f9410cceeb9 in _int_free (have_lock=0, p=0x7f940800cd70, av=0x7f9410f1ec40 <main_arena>) at malloc.c:4278 #5 __GI___libc_free (mem=0x7f940800cd80) at malloc.c:3124 #6 tcache_thread_shutdown () at malloc.c:2969 #7 arena_thread_freeres () at arena.c:950 #8 0x00007f9410ccf652 in __libc_thread_freeres () at thread-freeres.c:29 #9 0x00007f94121bb700 in start_thread (arg=0x7f940fd05700) at pthread_create.c:476 #10 0x00007f9410c5488f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
This could indicate a multi-threading issue with the creation of server and/or client in the test?