How a Kernel Bug Froze My Machine: Debugging an Async-profiler Deadlock

QuestDB is an open-source time-series database designed for high-performance workloads, offering features like ultra-low latency and high ingestion throughput, along with support for Parquet and SQL to maintain data portability. A developer, experienced in Linux systems, encountered a kernel bug when using the async-profiler to capture CPU heatmaps, which caused their machine to freeze due to a deadlock in the Linux kernel's handling of the cpu-clock event. The issue was traced to a recent update in the Ubuntu distribution using kernel version 6.17, which led to a deadlock during the hrtimer cancellation process in the perf_events subsystem. The developer explored the issue using QEMU for kernel debugging and identified the problem within the kernel's source code, eventually finding a workaround by using the -e ctimer option in async-profiler to avoid the problematic kernel feature. Despite this workaround, the developer delved deeper into the kernel's internals, using GDB to debug and manipulate the kernel state in a virtual environment, ultimately managing to unfreeze the machine by forcing the kernel to kill a process, showcasing both the complexity and intricacies of kernel-level debugging and the resilience required to solve such technical challenges.