Husky: Exactly-once ingestion and multi-tenancy at scale

Husky, Datadog's third-generation event store, uses a distributed, time-series oriented, columnar store optimized for streaming ingestion and hybrid analytical and search queries. Husky decouples storage and compute to scale independently. To ensure exactly-once ingestion of every event into Husky's storage engine, Husky Writers consume events from assigned shards and persist them with the goal of making them queryable as quickly as possible without introducing duplicates. Husky achieves this by using a hybrid approach to storing IDs in Husky itself, while also storing them in separate Husky tables for fast lookups. Additionally, Husky uses FoundationDB's strong transactional guarantees for conflict detection and resolution, ensuring that every piece of the ingestion path is stateless, scalable, and performant.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.