February 2022 Summaries

The ongoing enthusiasm for AI, machine learning (ML), and deep learning in IT operations and observability is driven by the complexity of modern systems and the hope for automation to ease operational burdens. While AI/ML offers potential benefits, such as reducing alert noise and aiding in root-cause analysis, there is a risk of overhyped expectations, especially when these technologies are used as black-box solutions without sufficient context or explainability. The current state of AIOps often integrates ML to enhance existing tools, but it can create additional work for teams lacking data science expertise. Tools like Observe aim to intelligently manage and curate machine data, focusing on automation and abstraction to simplify data correlation without relying heavily on AI/ML. This approach helps users navigate large volumes of data by providing structured datasets, auto-generated dashboards, and visualization tools that enhance understanding and efficiency. Ultimately, smarter tools are intended to augment human capabilities, allowing practitioners to focus on more valuable tasks in the face of increasing complexity and skills shortages in IT environments.

In 2017, the founders of Observe, backed by Sutter Hill Ventures, sought to revolutionize observability by creating a product that would seamlessly integrate diverse machine data types into actionable insights without requiring extensive pre-processing or complex tagging. They identified the limitations of existing observability solutions, which failed to break down data silos and left users with challenging data interpretation tasks. Opting for a buy-over-build strategy, Observe adopted Snowflake's platform, which met their requirements for handling semi-structured data and large-scale data ingestion efficiently, thanks to its separation of compute and storage resources. This integration allowed Observe to offer a flexible, cost-efficient pricing model, enabling users to store vast amounts of data cheaply and only pay when querying it, thus aligning with their goal of minimizing user constraints and maximizing data utility. Snowflake's capabilities in executing complex joins and managing large data volumes without incurring prohibitive costs were crucial for Observe to deliver a product that not only stores and manages data but also provides meaningful insights and context, liberating engineers from tedious data management tasks.

Observe, an observability platform, emphasizes the critical role of data in making sense of complex IT systems, aiming to address customer pain points through features like intelligent dataset curation and user-friendly navigation. The addition of an industry analyst to their team highlights their commitment to aligning product development with market needs, backed by a reputable leadership team including CEO Jeremy Burton. The platform leverages the Snowflake data platform to unify data and focuses on the "Event" as the core element of observability, catering to dynamic, distributed environments without creating significant data management challenges. Despite being a relatively new player, Observe has shown substantial progress in product development and customer adoption, underlining its potential in the evolving observability market. The company's forward momentum is complemented by insights from their 2021 State of Observability report, with plans to continue gathering valuable industry data and insights.

Richard Marcus, Head of Information Security at AuditBoard, discusses the challenges and strategies involved in managing a modern cloud-based risk management platform with a focus on observability. Observability is crucial for AuditBoard, a company that must comply with a wide range of standards and frameworks, as it helps ensure security, performance, and availability of their managed SaaS product. Marcus emphasizes the importance of collecting and retaining extensive data as a means of improving incident response and gaining customer insights, facilitated by Observe's commodity pricing model that allows for cost-effective data management. He shares his experience in using various tools for governance, risk, and compliance (GRC) initiatives and highlights the value of integrating observability data with customer support systems to proactively address potential issues. Future goals include exploring security-related applications, such as detecting anomalous data access, and automating threat detection processes to enhance their overall security architecture.