C++ Mutex Deadlock Detection with Callstack
February 18, 2014
There are tons of ways of going about debugging deadlocks in C++. This is an exercise and example of implementing a solution at the code level. This implements a special mutex type that detects deadlock based on a timeout and proceeds to print out the callstack of the offending threads. It allows you to troubleshoot and pinpoint where a deadlock occurs. The source can be found on GitHub.
$ ./callstack running thread 1 running thread 2 140602880931584140602872538880 Failed to lock in Failed to lock in 140602872538880140602880931584 locked by 140602880931584 locked by 140602880931584 Backtrace for 140602880931584 Backtrace for 140602880931584 _callstack_signal_handler, self: 140602880931584, target: 140602880931584, caller: 140602872538880 ./callstack(_Z15print_backtracem+0x5e) [0x401f80] ./callstack(_ZN6CMutex9timedlockERK8timespec+0x10d) [0x402421] ./callstack(_Z7thread1Pv+0x66) [0x4019ea] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fe0a931fe9a] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fe0a904c3fd] ./callstack() [0x401c7f] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7fe0a9327cb0] /lib/x86_64-linux-gnu/libgcc_s.so.1(+0xed6b) [0x7fe0a9543d6b] /lib/x86_64-linux-gnu/libgcc_s.so.1(+0xf490) [0x7fe0a9544490] /lib/x86_64-linux-gnu/libgcc_s.so.1(_Unwind_Backtrace+0x2e) [0x7fe0a9544d3e] /lib/x86_64-linux-gnu/libc.so.6(backtrace+0x58) [0x7fe0a9063108] ./callstack(_Z12GetCallstackmPPvi+0x4f) [0x401de4] ./callstack(_Z15print_backtracem+0x5e) [0x401f80] ./callstack(_ZN6CMutex9timedlockERK8timespec+0x10d) [0x402421] ./callstack(_Z7thread1Pv+0x66) [0x4019ea] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fe0a931fe9a] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fe0a904c3fd] Failed to lock in 140602880931584 locked by 140602880931584 Backtrace for 140602880931584 ./callstack(_Z15print_backtracem+0x5e) [0x401f80] ./callstack(_ZN6CMutex9timedlockERK8timespec+0x10d) [0x402421] ./callstack(_Z7thread1Pv+0x66) [0x4019ea] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fe0a931fe9a] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fe0a904c3fd] Failed to lock in 140602872538880 locked by 140602880931584 Backtrace for 140602880931584 _callstack_signal_handler, self: 140602880931584, target: 140602880931584, caller: 140602872538880 ./callstack() [0x401c7f] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7fe0a9327cb0] [0x7fff7d9ff707] [0x7fff7d9ff7e5] /lib/x86_64-linux-gnu/librt.so.1(clock_gettime+0x1d) [0x7fe0a9a4f15d] ./callstack(_ZN6CMutex9timedlockERK8timespec+0x5a) [0x40236e] ./callstack(_Z7thread1Pv+0x66) [0x4019ea] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fe0a931fe9a] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fe0a904c3fd]