How we built an async-aware Python profiler

Datadog redesigned its Python profiler to better analyze asyncio applications, where traditional flame graphs treat concurrent tasks as separate stacks and obscure the application paths responsible for work. The new async-aware model reconstructs task dependency relationships, creating “stacked stacks” that attribute activity such as database queries to the tasks awaiting them while accommodating tasks that outlive their creators. Because tracking every asyncio task substantially increased sampling costs, the team also optimized the profiler by removing exceptions from the sampling path, interning repeated strings, and replacing costly process_vm_readv memory reads with protected memcpy operations that safely recover from rare memory-access faults. These changes reduced profiler overhead by more than 60% in deployed services, enabled higher sampling fidelity, and helped identify production issues including an infinite decompression loop, wasteful logging string construction, and inefficient numerical code. The async-aware profiling and lower-overhead improvements are available in recent ddtrace versions.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.