February 2023 Summaries

Multicloud observability is a concept that emerges from the increasing adoption of multiple public cloud services by organizations to enjoy the benefits of diverse innovations and to mitigate risks like outages. While multicloud usage offers access to unique proprietary services and enhanced reliability, it also presents challenges such as operational complexity, security, and compliance issues. Observability, distinct from mere monitoring, provides a comprehensive view across these disparate cloud environments, aiding in prompt issue detection and resolution. It involves unifying data from various clouds into a single, coherent framework, thereby avoiding silos and facilitating better decision-making. Tools such as Observe's Observability Cloud offer solutions by integrating data from major cloud providers like AWS, GCP, and Azure into a single platform, which enhances operational efficiency and reduces costs through a usage-based pricing model. This approach allows organizations to manage vast amounts of data effectively without the financial burden typically associated with traditional data observability methods.

Observe has introduced a feature called Ingest Monitoring to enhance error visibility during data ingestion by automatically forwarding any errors to the System Datastream. This new feature simplifies the process of identifying and troubleshooting ingest errors by allowing users to easily navigate through their workspace settings to view the latest errors, complete with detailed information such as the source datastream, endpoint, request time, and content type. By setting the X-Request-Id header on ingest requests, users can quickly correlate and identify the source of an error. The platform also provides additional resources for troubleshooting data ingestion, including a dedicated documentation page and support via Slack.

Observe built the Observability Cloud on Snowflake to leverage its ability to handle vast amounts of data with efficiency and scalability, offering resiliency and global reach. The platform ingests data from various customer applications using open endpoints and agents, storing it in a Data Lake to create a Data Graph that allows comprehensive exploration and monitoring of organizational data. Snowflake's separation of storage and compute resources enables Observe to scale efficiently, accommodating large workloads with flexible, on-demand compute resources. Observe's micro-batch ingest process smooths unpredictable data streams into manageable chunks, optimizing data transfer to Snowflake and reducing costs. The system employs Kafka for buffering, converting data into the "observation" format, and utilizing Amazon S3 for storage before loading into Snowflake. This approach enhances data load efficiency and cost-effectiveness, allowing Observe to provide savings to customers. The strong partnership with Snowflake facilitates collaborative improvements benefiting both companies and their users.