February 2025 Summaries

Grafana Mimir, an open-source, horizontally scalable time series database (TSDB), is marking its third anniversary as a leading Prometheus-compatible metrics backend with over 4.6k GitHub stars. As part of its evolution, Grafana is making architectural changes to Mimir and Grafana Cloud Metrics to enhance reliability and support future growth. The current architecture, which involves stateful ingester nodes with in-memory TSDB and a write-ahead log (WAL), poses challenges such as disrupted writes by heavy queries and complex management. To address these, Grafana is redesigning Mimir by decoupling the read and write paths using Apache Kafka, and for Grafana Cloud Metrics, exploring WarpStream. WarpStream is a Kafka-compatible data streaming platform built on object storage, offering cost-effective multi-availability zone deployments without inter-AZ networking costs and providing stateless, auto-scaling benefits. Load tests confirmed WarpStream's capability to handle Grafana Cloud Metrics' scalability needs, and the new architecture is gradually being rolled out, promising enhanced resilience to traffic spikes and ingest volume. Once the rollout is stable, the new architecture will be fully implemented in Mimir's open-source version.

Character.AI, a company with over 20 million monthly active users, has refined its data management approach to enhance operational efficiency and reduce costs. Initially relying on a batch-oriented ETL process using DBT, the company shifted to real-time data processing to gain better user insights. This transition required a scalable solution, leading to the adoption of WarpStream, which acts as a Kafka-compliant facade on top of an Object Store bucket. WarpStream's stateless agent architecture, integration with Google Cloud Storage, and Managed Data Pipelines powered by the open-source Bento framework allowed Character.AI to manage real-time data effectively without additional infrastructure. This setup enabled the company to stream over 3 GiBs of data per second into its data warehouse and reduce analytics costs by building a scalable, near-real-time analytics engine. Character.AI's evolution in data management demonstrates its commitment to leveraging innovative technologies to improve user experience and operational capabilities while remaining cost-effective.

WarpStream has introduced Audit Logs to provide a comprehensive and structured record of all authentication actions, authorization decisions, and platform operations across Kafka clusters, addressing the challenges of limited built-in auditing in Kafka. This feature is crucial for compliance and enhances incident response and root-cause analysis by offering a centralized view of operations, with logs that are structured, queryable, and compatible with existing tools. Audit Logs capture two categories of events: Cluster Audit Logs for Kafka-level operations and Platform Audit Logs for account-level operations, following the CloudEvents spec for seamless integration. These logs are produced into a fully-managed WarpStream cluster, allowing consumption via the Kafka protocol and integration with various systems like SIEMs or data lakes, enhancing both audit trail visibility and flexibility. Each log event is a structured JSON payload detailing the action, actor, timing, and result, with all metadata retained without egressing data from WarpStream clusters. The pricing model is based on data written, storage, and network usage, with most use cases remaining cost-effective. Audit Logs are now available for all WarpStream Pro or Enterprise accounts, requiring minimal setup and offering immediate data flow upon activation.

WarpStream is a data streaming platform compatible with the Apache Kafka® protocol, designed to leverage cloud object storage for cost-efficient and durable data handling, albeit with a tradeoff in latency. The platform has introduced Lightning Topics, which, when used with S3 Express One Zone (S3EOZ), reduce median produce latency to 33ms and p99 to 50ms without additional costs, by decoupling the durability and sequencing processes. This approach, inspired by the LazyLog paper, allows for asynchronous commitment to the Control Plane, resulting in faster acknowledgments to producers albeit at the cost of no returned offsets, idempotent producers, or transactions. To further ensure data reliability, WarpStream employs a 'Slow Path' to guarantee zero data loss by periodically checking and committing journaled files. Additionally, the new Ripcord Mode enables WarpStream Agents to continue processing produce requests even when the Control Plane is unavailable, although it cannot perform operations requiring Control Plane metadata or start new Agents under such conditions. Both Lightning Topics and Ripcord Mode are now generally available for use.