Observability Scale: Scaling Ingest to One Petabyte Per Day

Observe offers a data observability platform designed to handle vast amounts of telemetry data, allowing customers to monitor and introspect their systems without needing to sample data, thus supporting up to a petabyte of data ingestion per day for a single tenant. The platform's architecture involves an ingest pipeline that processes data through various stages, including load balancing, data validation, and storage in Snowflake databases, all while ensuring high throughput and cost-efficiency. Initially, Observe struggled with memory and processing challenges as it scaled from handling 30 TB/day to 200 TB/day, but by optimizing their system through changes like converting certain components from Golang to C++ and adopting a parallel datastream insertion strategy, they succeeded in reducing costs and latencies. Further improvements involved addressing Kafka's partition limits and implementing a topic-per-datastream approach for high-volume customers, enabling the platform to manage the ingest of 1 PB/day. Future enhancements are planned, such as utilizing Snowpipe Streaming and refining internal data formats, with the aim of making the system even more efficient and scalable.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.