March 2026 Summaries

The blog post discusses a tutorial on setting up observability for a Kubernetes workload using groundcover, a tool that provides HTTP trace visibility without needing to modify application code. By leveraging eBPF technology, groundcover instruments the cluster at the Linux kernel level, automatically generating traces, metrics, and service maps without the need for SDK installs or other complex configurations. This approach contrasts with traditional observability models that require code changes and sampling decisions, often leading to blind spots and cost concerns. The tutorial guides users through the process of deploying groundcover on an EKS cluster and emphasizes the advantages of this method, such as stable data collection, simpler pricing based on nodes rather than data volume, and enhanced flexibility in data handling. Additionally, groundcover is compatible with existing telemetry systems like OpenTelemetry or Prometheus, allowing users to integrate new and existing data seamlessly.

In a critical analysis of current trends in observability platforms, Shahar Azulay argues against the prevalent approach of adding AI features as an overlay to existing systems, highlighting the limitations such an approach inherits from the underlying platform. Azulay emphasizes the importance of integrating AI natively within the platform to enhance its capabilities, specifically advocating for the use of eBPF sensors that capture telemetry data automatically, without the need for manual instrumentation. This method ensures comprehensive data visibility and control, allowing the AI agent to create meaningful assets like dashboards and monitors within the user's environment. Azulay also underscores the significance of maintaining data sovereignty, ensuring that sensitive information remains within the customer's control, which avoids potential security concerns. The article concludes by acknowledging ongoing challenges in refining the platform's capabilities, such as memory retention and dashboard creation, but maintains that the foundational architecture is crucial for effective AI-powered observability.

Groundcover has redefined the Bring Your Own Cloud (BYOC) deployment model by making it the sole method of operation, offering a self-serve system that allows any team to deploy their observability platform in their own cloud accounts, such as AWS or GCP, without human intervention. This innovative approach contrasts with traditional SaaS models that often complicate BYOC deployment, which typically involves extensive manual setup and professional services. Groundcover's solution enables customers to begin using the full product in minutes, providing a 14-day trial without feature limitations to test the platform using their actual production infrastructure. After the trial, a free tier with 12-hour data retention remains available, and paid plans are offered with flexible pricing based on monthly average host count, which includes options for expanded data retention and support. The company's pricing strategy, which does not depend on data volume, aims to align with customers' efficiency needs, making it a unique player in the observability space.

Groundcover has launched Session Replay as part of its Real User Monitoring (RUM) feature, allowing users to record and review real user sessions directly within the Sessions Explorer while keeping all data stored in the user's environment to maintain privacy and control. This new feature addresses the limitations of traditional RUM by providing visual playback of user interactions, including clicks and scrolls, which can be cross-referenced with event timelines and backend traces, offering a comprehensive view of user experiences. The development of Session Replay was driven by the need to bridge gaps in understanding user sessions when logs and traces are insufficient, and the approach taken ensures that user data remains secure by avoiding third-party cloud storage. Future enhancements are planned to improve filtering, direct trace-to-replay navigation, and player functionalities. Session Replay is available at no additional cost for current RUM users, providing an integrated solution for monitoring frontend and backend interactions.

As AI-generated code rapidly increases, traditional observability methods face significant challenges, particularly in handling the unique log shapes and field conventions that AI services introduce. The foundational issue is that conventional tracing, designed for request/response cycles with clear latency and error signals, does not align well with the long reasoning loops and unique error handling of LLM agents. While eBPF offers network-level visibility irrespective of code origin, a new approach is necessary for assessing AI agent behavior, which requires adaptive, content-aware sampling rather than static, manually maintained pipelines. This shift emphasizes the importance of semantic content, such as prompts and decisions, over traditional metrics like latency. OpenTelemetry SDKs and evolving conventions are adapting to these needs, aiming to integrate with existing infrastructures rather than rebuild them. However, the economic pressures of telemetry volume can counteract these efforts, underlining the relevance of solutions like groundcover's BYOC architecture, which decouples telemetry cost from volume, allowing for deeper, more meaningful instrumentation without financial constraint.

Groundcover faced challenges with Keep, an open-source notification engine, due to its complexity and inefficiencies at scale, leading to operational headaches and difficulty in debugging issues. As Keep was no longer meeting their needs, Groundcover developed their own system, the Dispatch Center, leveraging the Temporal open-source workflow engine to manage notification routing with gcQL instead of complex YAML workflows. This new system allows for transparent configuration, live previews, and simulation of alerts, offering better control and diagnostics. The Dispatch Center is managed through Terraform, allowing configurations to live in version control, and it integrates seamlessly with Groundcover's existing infrastructure. While running alongside Keep during the transition, the Dispatch Center is poised to enable future automation and AI-driven notification management, transforming the notification layer from a limiting factor into a strategic asset.

Groundcover's Synthetic Performance Monitoring addresses the limitations of reactive monitoring by running scheduled checks that simulate real requests against endpoints, alerting teams immediately when issues occur, thereby eliminating blind spots such as low-traffic APIs and off-hours failures. Unlike traditional synthetic tools that operate from external data centers, groundcover's solution operates from within a user's own cloud environment, ensuring that monitoring can include internal services and that request data remains private. The system uses multi-layered assertion logic to detect issues beyond simple status codes, such as broken JSON bodies or slow response times, and integrates seamlessly with existing observability stacks by generating distributed traces alongside real application traces. This approach ensures comprehensive monitoring without additional licensing costs, as it is included in groundcover's Pro, Enterprise, and On-Prem plans, and allows users to quickly set up tests and route alerts through familiar channels like PagerDuty and Slack.

In the realm of observability, teams often prioritize logging due to its familiarity and flexibility, but this can lead to an overreliance that creates noise and strains backend resources. Metrics naturally follow as teams seek scalable solutions for measuring trends, while tracing, essential for understanding the "why" behind system behaviors, is often deferred due to its complexity and the coordination required across services. Manual instrumentation for tracing is frequently deprioritized, but auto-instrumentation SDKs like OpenTelemetry offer significant improvements, albeit with limitations such as added overhead and incomplete framework support. eBPF (Extended Berkeley Packet Filter) addresses some of these challenges by providing kernel-level visibility without prior instrumentation, capturing real-time network interactions and enriching OpenTelemetry traces with detailed data such as full payloads and HTTP headers. The combination of eBPF and OpenTelemetry is advocated as a comprehensive observability solution, with eBPF handling network-level coverage and OpenTelemetry SDKs offering application-level insights, thereby overcoming the traditional gaps in observability by integrating both approaches for full visibility from network to application logic.

A recent panel discussion explored the concept of Bring Your Own Cloud (BYOC), which involves deploying vendor software within a customer's cloud account rather than the vendor's SaaS environment. This model is gaining traction due to its ability to meet data sovereignty and regulatory requirements, provide cost transparency, and align with enterprise infrastructure needs. Companies like ClickHouse, Zilliz, and groundcover are adopting BYOC with varying approaches, focusing on strong separation between control and data planes, privacy, and cost alignment. While BYOC offers benefits such as reduced egress costs and enhanced data control, it also introduces operational complexities for vendors, requiring robust automation and cloud-native tools. The rise in AI workloads is further accelerating BYOC adoption, as it allows organizations to handle increased data volumes more efficiently. Despite its advantages, BYOC is not a universal replacement for SaaS, and organizations without strict data residency needs may still find SaaS to be the simpler option.

Container image signing is a crucial security measure that enables automatic verification of an image's integrity and authenticity by ensuring it hasn't been tampered with and was signed by an approved identity before deployment. It does not, however, guarantee the absence of vulnerabilities, necessitating its combination with scanning, attestations, and runtime monitoring for comprehensive protection. Effective implementation involves integrating signing into CI/CD pipelines, storing signatures within OCI registries, and enforcing verification at Kubernetes admission controls, with a preference for signing immutable digests over mutable tags to prevent potential loopholes. Various trust models, including central key management, keyless OIDC-based signing, and private PKI, determine the sustainability and security of the signing program, while tools like Sigstore Cosign and Kyverno support the end-to-end signing process. Despite the advantages, challenges such as ensuring continued signature availability and managing trust policy drift persist, underscoring the importance of runtime visibility and monitoring to detect anomalies post-deployment.