Company
Date Published
Author
Richard Artoul, Cecilia Watt
Word count
2680
Language
English
Hacker News points
191

Summary

The story of "Husky", a new event storage system built by Datadog, revolves around the need to improve upon their existing Log Management product. The old system had limitations in handling log data, such as pre-aggregating many similar events into a single aggregated datapoint, which made it difficult to store and query log data efficiently. To address these issues, the team decoupled storage from clustering and introduced new roles for writers, readers, and compactors, each serving distinct purposes. The system is designed to be highly scalable, flexible, and performant in the face of unpredictable and aggressive multi-tenant workloads. Husky's architecture separates compute from storage, allowing for better control over isolation, performance, and quality of service in a highly multi-tenant environment. The system has improved reliability, performance, and efficiency compared to its predecessor, with reduced latency and increased flexibility in how queries are isolated and allocated costs.