guacd segfaults when user actively presses keys at RDP disconnect time

Program terminated with signal 11, Segmentation fault.
#0  guac_rdp_keyboard_update_keysym (keyboard=0x7faa4c364010, keysym=65288, pressed=0) at keyboard.c:444
444         if (!keyboard->synchronized) {

(gdb) p keyboard->synchronized
Cannot access memory at address 0x7faa4c36401c

(gdb) bt
#0  guac_rdp_keyboard_update_keysym (keyboard=0x7faa4c364010, keysym=65288, pressed=0) at keyboard.c:444
#1  0x00007faae2f50bf0 in guac_user_input_thread (data=<optimized out>) at user-handshake.c:165
#2  0x00007faae2372ea5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007faae0c678cd in clone () from /lib64/libc.so.6

(gdb) info threads
  Id   Target Id         Frame
  3    Thread 0x7fa9eb7fe700 (LWP 7379) 0x00007faae2374017 in pthread_join () from /lib64/libpthread.so.0
  2    Thread 0x7fa9ebfff700 (LWP 7376) (Exiting) 0x00007faae2379bad in recvmsg () from /lib64/libpthread.so.0
* 1    Thread 0x7fa9f8ff9700 (LWP 7381) guac_rdp_keyboard_update_keysym (keyboard=0x7faa4c364010, keysym=65288, pressed=0)
    at keyboard.c:444

"info threads" shows that there are only 3 threads running:

1. guac_user_input_thread (the thread segfault occured in)
2. guacd_connection_thread
3. guacd_user_thread waiting for guac_user_input_thread

Note that there is no client thread running.
So here is what happenned: guac_rdp_client_thread exited and freed rdp_client->keyboard structure, while user input thread was still running and tried to access rdp_client->keyboard after it was freed.

This is an old bug. Found a 2-year old GUACAMOLE-433 which looks exactly the same, but it was closed as "Cannot reproduce". It does not happen often indeed. We have 120 users actively using Guacamole (since COVID-19) and have seen this segfault only twice within the last month.

After some research I managed to reliably reproduce it.

It is reproducable with 1.1.0, staging/1.2.0 and master branches.

Steps to reproduce:

1. Connect to RDP server
2. Block connnection to RDP server with iptables (or create conditions for disconnect in any other way):

   iptables -A OUTPUT -d <RDP_SERVER_ADDRESS> -j REJECT

3. Use keyboard or mouse actively. The easiest way is to push some key and hold it down for 30-60 seconds.
4. Segmentation fault will happen in next 30-60 seconds at RDP disconnect time.
5. Release the key you kept pressing

Segmentation fault can happen in multiple places depending on what you were doing at the time of disconnect, but typically it happens in keyboard or mouse handlers.
Since it is a race between threads, you may have to make several attempts to reproduce it.
I found it easily reproducable on a VM with a single CPU core, but on a physical machine with 24 CPU cores I had to make 3 or 4 attempts.