March 2023 Summaries

The Observability Cloud introduces a revolutionary architecture that transforms how observability data is ingested, stored, analyzed, and visualized, centering around three main components: the Data Lake, Data Graph, and Data Apps. The Data Lake serves as a unified repository for various telemetry types, from metrics and logs to traces and event data, utilizing open-source collectors like Telegraf and Fluentbit to avoid vendor lock-in. Built on Snowflake, the Data Lake allows for scalable storage and analysis without the constraints of data silos or fixed schemas, offering cost-effective long-term storage by compressing data significantly. This setup supports a usage-based pricing model, making observability practices economically scalable and eliminating the need for tough decisions about data retention. The system's design ensures rapid data processing and contextual insights during troubleshooting, paving the way for the subsequent role of the Data Graph in further enhancing data linkage and relevance.

Observe leverages Snowflake's efficient, compressed columnar data storage to achieve significant data compression, often reaching 10x the original data size, as part of its Observability Cloud's unique architecture. The platform integrates data from various sources into a Data Lake, transforming it into an interactive Data Graph that enables comprehensive data exploration and monitoring. Using the custom OPAL language, which is more straightforward than SQL, Observe models and transforms data into Datasets that represent both business and infrastructure elements, such as user sessions and Kubernetes pods. These Datasets are classified into Event, Resource, Interval, and Table types, each serving different analytical purposes. Observe compiles OPAL into SQL for use with Snowflake, where Datasets are stored in point or interval tables, optimized for efficient querying and cost-effective storage. Datasets can be accelerated for faster querying based on likely access patterns, with a flexible retention policy, allowing businesses to tailor data retention to their needs. This setup enhances data reliability and performance without imposing limits on data storage capacity, paving the way for the next part of the blog series, which will focus on resource management in terms of resilience, quality, and cost.

Microsoft Azure's built-in monitoring tools provide a basic foundation for cloud observability, but to achieve a comprehensive and effective observability strategy, additional solutions like the Observe Azure App are recommended. Azure, as a rapidly growing cloud provider, offers various services such as Azure Virtual Machines and Azure Kubernetes Service, along with native tools like Azure Monitor and Application Insights for monitoring and application performance management. However, these tools have limitations, including lack of multi-cloud support, data silos, and limited customization, which can hinder performance and reliability insights. The Observe Azure App addresses these challenges by offering multi-cloud observability, seamless data ingestion, cost-effective storage, and meaningful data correlation, enabling businesses to gain deeper insights and maintain a robust observability strategy across their entire IT infrastructure.

The Observability Cloud is a comprehensive solution designed to streamline the management of observability data by consolidating all data types into a single platform, eliminating the need for multiple tools and reducing complexity in troubleshooting distributed applications. It employs a novel architecture comprising a Data Lake for unified telemetry storage, a Data Graph to map and connect relevant datasets for quick context retrieval, and Data Apps that facilitate data ingestion and provide pre-built integrations, dashboards, and monitors for popular services like AWS, Azure, and Kubernetes. This architecture supports a usage-based pricing model, ensuring scalability and cost-effectiveness as data grows. The Observability Cloud leverages open-source collectors for data ingestion, compresses data for long-term storage, and separates storage from compute using Amazon S3 and Snowflake, enhancing both economic efficiency and performance. By offering a visual representation of datasets and their relationships through the Data Graph, it enables faster querying and incident investigation, providing organizations with a robust, SaaS-based solution to manage their observability needs efficiently.