June 2023 Summaries

At KubeCon Europe 2023, a significant discussion emerged about the need for a standardized query language in observability to address the challenges posed by the diversity of existing query languages. This initiative, driven by leaders from eBay and Netflix, has led to the formation of the Observability Query Standard Working Group under the Cloud Native Computing Foundation (CNCF). The group's goal is to develop a unified query language that facilitates vendor-agnostic interoperability and streamlines querying across different observability data types like logs, metrics, and traces. The diverse landscape of query languages poses a significant challenge for DevOps professionals, as it complicates integration and consistency across systems, making it difficult for organizations to adapt to technological changes. The working group, still in its early stages, seeks to gather insights from the open-source community to create a comprehensive database for recommendations, much like SQL's role in databases, with regular meetings and collaborative efforts open to interested individuals.

Logz.io, founded by Tomer Levy and Asaf in 2015, initially aimed to enhance log management using the ELK Stack but soon recognized that logs alone were insufficient for comprehensive system visibility. To address this, they developed Logz.io Open 360™, an observability platform that unifies log, metric, and trace data, offering a more complete view of production environments while focusing on cost-efficiency. The platform utilizes data optimization to reduce unnecessary telemetry data, integrates open-source technologies for interoperability and cost savings, and introduces tools like LogMetrics to convert logs into metrics, thereby lowering costs. Additionally, Logz.io introduces Cold Search for cost-effective log search and invites users to explore their AI-powered observability solutions with a free 14-day trial.

Logz.io's Cloud SIEM has become a key component of the new AWS AppFabric, enabling seamless integration with other SaaS applications for AWS customers. This collaboration is designed to enhance IT and security operations by providing a standardized schema that facilitates task automation and improves productivity through generative AI. The integration aims to eliminate the complexities and costs associated with point-to-point integrations, thus improving cross-application workflows and security observability. Logz.io Cloud SIEM offers a cloud-native, fully managed solution that rapidly analyzes security data, empowering lean security teams to efficiently manage threats while maintaining high performance across distributed environments. Built on open-source technology with deep integrations into AWS solutions such as Amazon OpenSearch, this partnership allows customers to leverage existing skills and rapidly gain insights by running queries and maintaining data availability across Amazon S3 storage.

Log management has evolved significantly, especially with the transition from the open-sourced ELK Stack to AWS's OpenSearch, following Elastic's decision to close source its toolset. OpenSearch Dashboards, which closely resembles Kibana, serves as an integral tool for querying and visualizing log data. Effective log analysis requires proper log parsing, often facilitated by tools like FluentBit or services like Logz.io's parsing-as-a-service, which transforms raw log data into searchable fields. Once parsed, logs can be queried using OpenSearch Dashboards, which allows for building complex queries with the Dashboard Query Language (DQL) and visualizing data trends through various graphical representations. Logz.io enhances these capabilities by integrating full observability, alerting, and AI-driven insights, offering a comprehensive SaaS platform that combines log, metric, and trace analytics to streamline the troubleshooting process and improve mean time to resolution (MTTR). As businesses grow and log volumes increase, these tools are critical for maintaining efficient log management and analysis.

OpenTelemetry (OTEL) is an open-source observability platform under the Cloud Native Computing Foundation (CNCF) designed to provide a standardized, vendor-agnostic method for instrumenting, collecting, and exporting telemetry data, including traces, logs, and metrics, across various programming languages, platforms, and cloud environments. Formed by the merger of OpenTracing and OpenCensus in 2019, OpenTelemetry has rapidly gained traction, becoming the second most active CNCF project after Kubernetes. It allows organizations to reduce the number of tools needed for telemetry data management, with major cloud vendors like AWS and Microsoft Azure offering their own distributions. A case study involving Logz.io demonstrates how OpenTelemetry can be used to collect and analyze telemetry data through client libraries and the OpenTelemetry Collector, which can receive and export data using multiple protocols. By adopting OpenTelemetry, organizations can automate tasks, minimize human errors, and ensure data is in a standard format compatible with various third-party applications and services, thereby enhancing their troubleshooting efficiency and application performance monitoring.

Logging as a Service (LaaS) offers a cloud-native solution for managing log data, enabling organizations to focus more on deriving value from their data and less on infrastructure management. This approach simplifies log management by outsourcing scaling and maintenance tasks, allowing for seamless data handling even as volumes fluctuate. While self-hosted solutions provide greater control, they demand more resources and technical expertise, often becoming challenging as data needs grow. LaaS platforms, such as Logz.io, streamline data collection and analysis by providing automated data correlation and visualization capabilities, ultimately reducing the mean time to resolution for production issues. These platforms facilitate a unified view of telemetry data, enhancing observability and troubleshooting efficiency. Additionally, LaaS solutions offer cost optimization features, such as data usage tracking and customizable storage options, making them attractive for organizations looking to manage costs while maintaining robust log management capabilities.

Open standards and open specifications are increasingly vital in the fragmented and polyglot landscape of modern IT systems, driving compatibility, collaboration, and convergence within the observability domain. Significant updates from KubeCon highlight efforts to standardize various protocols and systems, such as the formation of a new working group for an Observability Query Language Standard by eBay and Netflix, and the formalization of the Prometheus Remote-Write Protocol as an IETF open standard to enhance interoperability. Additionally, the transition from OpenCensus to OpenTelemetry, which now offers feature parity across multiple languages, and the merger of the Elastic Common Schema with OpenTelemetry, reflect a move towards unified frameworks that simplify workflows and enhance compatibility. These endeavors, along with the stabilization of the OpenTelemetry Protocol and Logs, underscore a shift towards open-source-based specifications that foster innovation and community collaboration, ultimately enriching the observability ecosystem for practitioners and developers.

OpenObservability Talks celebrated its third anniversary with a return to in-person events and conferences, including a special live episode from the KubeCon event. Throughout the year, the podcast explored various aspects of observability, focusing on challenges associated with monitoring Kubernetes environments and cloud-native technologies, as well as insights from major tech companies like Google and Meta. Prominent projects such as Prometheus and OpenTelemetry were discussed in detail, alongside emerging tools like Backstage and OpenCost. The show highlighted the increasing importance of observability in areas like Platform Engineering and FinOps, with industry experts sharing their experiences and best practices. As Dotan Horovits, the host, was nominated as a CNCF Ambassador, the podcast also served as a media partner for CNCF, providing exclusive insights from its events and leaders. The upcoming fourth year promises more engaging discussions, including a focus on eBay's observability practices, and the introduction of a new CNCF working group on query language standards for observability. The show remains accessible through various platforms and encourages audience interaction during live streams.

Security Information and Event Management (SIEM) systems have been integral to security infrastructures for nearly two decades, yet there remains a gap in guidance for effectively evaluating and managing these solutions. The blog highlights that many SIEM vendors exploit this knowledge gap by not providing adequate value to customers. To address this, the article offers insights drawn from the authors' experiences as both buyers and sellers of SIEM software, aiming to help organizations save costs, streamline operations, and foster successful vendor partnerships. Key recommendations include tailoring SIEM solutions to meet specific organizational needs, delving deeply into technical details to ensure system transparency and efficacy, and being mindful of budgetary constraints and vendor negotiation strategies. Furthermore, it emphasizes the importance of considering future integrations with other technologies and the necessity of optimizing data sources to enhance cost efficiency and investigative effectiveness.

Log parsing, though often overlooked, is essential for effective log analysis by extracting and organizing information from logs into structured fields. The text compares several popular technologies for log file parsing, emphasizing that these tools not only parse logs but also handle log collection, processing, storage, and analysis. Logz.io provides an easy-to-use platform with automatic parsing for common log types and a self-service parser for custom needs, making it ideal for low-maintenance scenarios. Fluentd, an open-source tool with strong Kubernetes integration, offers community-developed plugins for flexible log manipulation, while Logstash, part of the ELK Stack, is seen as complex and less favored compared to newer options. Vector, maintained by Datadog, uses a simpler expression-oriented language for log parsing, and Cribl specializes in advanced data pipeline management with a focus on reducing vendor lock-in and cost. Each tool has unique strengths, making the choice dependent on specific needs such as integration with existing systems, cost considerations, and the complexity of the desired observability pipeline.