Company
Date Published
Author
Khayyam Guliyev, Duarte Nunes, Ming Chen, Justin Jaffray
Word count
4682
Language
English
Hacker News points
None

Summary

Datadog has developed a sixth-generation real-time timeseries database, designed in Rust, to handle growing data volumes, complex queries, and high-cardinality workloads efficiently. This new system, named Monocle, employs a log-structured merge tree (LSM tree) architecture, enabling high throughput and low latency while utilizing a shard-per-core model to manage data ingestion and storage. The evolution from previous systems like Cassandra, Redis, MDBM, and a Go-based B+ tree to this unified Rust-based engine has enhanced performance, offering a 60x increase in ingestion speed and 5x faster query execution. Key design features include sharded ingestion pipelines, a shared radix-tree buffer for memory-efficient aggregation, and a unified series cache to reduce query latency. The architecture's modular nature allows for reuse across Datadog's infrastructure, promoting consistency and reliability. Future improvements aim at dynamic load balancing and potentially unifying indexing and timeseries storage to further adapt to Datadog's growing demands and customer needs.