Company
Date Published
Author
Daniel Intskirveli, Cecilia Watt
Word count
4375
Language
English
Hacker News points
22

Summary

Husky, Datadog's third-generation event store, uses a distributed, time-series oriented, columnar store optimized for streaming ingestion and hybrid analytical and search queries. Husky decouples storage and compute to scale independently. To ensure exactly-once ingestion of every event into Husky's storage engine, Husky Writers consume events from assigned shards and persist them with the goal of making them queryable as quickly as possible without introducing duplicates. Husky achieves this by using a hybrid approach to storing IDs in Husky itself, while also storing them in separate Husky tables for fast lookups. Additionally, Husky uses FoundationDB's strong transactional guarantees for conflict detection and resolution, ensuring that every piece of the ingestion path is stateless, scalable, and performant.