"Always-on" production memory profiling in just 5 instructions

Redpanda utilizes the C++ Seastar framework, which employs a thread-per-core architecture that enhances memory allocation efficiency by overriding default methods, although it limits the use of virtual memory and existing analysis tools. To address memory usage analysis, Seastar introduced a sampled heap memory profiler that operates with minimal overhead, allowing continuous use in production environments to debug real-life workload issues. This profiler records only a fraction of allocations, using a dynamic, randomized sampling interval to prevent biases, and tracks live allocations to identify the most significant memory consumers. Small and large allocations are managed differently, with small ones grouped into pools and large ones tracked via page metadata, allowing the profiler to attribute memory usage to specific code sites through backtraces. The profiler's integration into Seastar enables real-time memory usage monitoring through an HTTP API, although it's particularly useful during out-of-memory scenarios by identifying top memory allocation sites. Despite added profiling overhead, optimizations to the Seastar allocator improved overall performance, with the profiler proving instrumental in diagnosing and resolving memory issues efficiently, and the enhancements have been merged into upstream Seastar for broader community benefit.