Inside Husky's query engine: Real-time access to 100 trillion events

Datadog's third-generation event store, Husky, is designed to process over 100 trillion daily events and enable interactive querying at scale by overcoming challenges like varying data schemas and massive data volumes. The architecture comprises a multi-tenant query engine that includes services like the query planner, orchestrator, metadata service, and reader service, which work together to optimize and execute queries efficiently. Husky employs advanced techniques such as fragment pruning, lazy evaluation, multi-layered caching, and shuffle sharding to ensure performance, reduce computational load, and maintain tenant isolation. These strategies allow Husky to handle trillions of queries with sub-second interactivity, scanning a minimal portion of the data. Husky continues to evolve by adopting modularity and interoperability standards like Apache Arrow and Parquet to enhance system adaptability and integration, with plans to further decouple query compute from cache storage to improve efficiency and scalability.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.