July 2025 Summaries

The Aiven blog article delves into the performance characteristics of Diskless Kafka topics, a new approach in Apache Kafka that stores data directly in object storage like AWS S3, offering significant cost savings and limitless retention with strong durability. This method flips the traditional Kafka architecture by bypassing local disk storage and using a batch coordinator, such as PostgreSQL, for metadata management. While this setup achieves extreme durability and reduces operational complexity, it introduces latency due to the need for batching and object storage uploads, making it more suited for high-throughput workloads. The article emphasizes that while Diskless Kafka topics offer compelling advantages, including a potential 80% reduction in the total cost of ownership, they are not direct replacements for traditional Kafka topics due to their unique performance trade-offs. Performance testing reveals challenges with single producer scenarios, but scaling with multiple producers and partitions can mitigate latency spikes. The article also highlights the importance of optimizing the batch coordinator, especially as partition counts increase, and provides tuning recommendations for various deployment scenarios. As Diskless Kafka topics continue to evolve, understanding their trade-offs is crucial for making informed architectural decisions.

Apache Kafka and Apache Iceberg face challenges in integration due to the costly and complex process required to convert Kafka messages into Iceberg tables. This process involves multiple stages, resulting in high expenses and operational burdens. The current solutions, such as community connectors or proprietary services, either lack robustness or result in vendor lock-in, leading to inefficiencies and high costs. The article argues for a unified approach that would allow Kafka topics to natively become Iceberg tables without the need for additional connectors or storage, creating a seamless and cost-effective integration. This vision aligns with the growing trend towards using cloud object stores and aims to leverage Kafka's extensive user base to drive the adoption of this more streamlined, open-source solution. The ultimate goal is to eliminate the fragmented landscape of proprietary solutions, reduce costs, and enhance the accessibility and functionality of real-time data processing.

Aiven's blog post discusses the company's new approach to stream governance for Kafka data streams, emphasizing a developer-centric philosophy that aims to make reliable data streaming the default, rather than a costly add-on. The blog introduces Aiven Stream Governance, which provides a streamlined, topic-centric user experience, allowing developers to manage schemas, connectors, and permissions from one interface, significantly reducing the setup time from 20 minutes to 30 seconds. This approach addresses the challenges and frustrations developers face with Kafka's complex configurations, which often result in a high cognitive burden and technical debt. Aiven's goal is to make data streaming intuitive and efficient, embedding governance as a natural part of the development process without sacrificing speed or reliability, and offering these tools for free to ensure that data streaming is accessible to all.

Diskless Kafka architecture offers a transformative approach to Apache Kafka deployments by leveraging object storage to eliminate costly inter-AZ egress charges associated with traditional synchronous replication. This architecture allows data to be replicated directly into S3 or similar object stores, providing significant cost savings and on-demand elasticity despite object storage's slower performance compared to local SSDs. While it introduces new trade-offs in durability and latency, diskless Kafka enables a separation of compute and storage layers, allowing for more flexible scaling and efficient resource utilization. It supports various workloads, from low-latency transactional systems to cost-efficient log and telemetry aggregation, by enabling hybrid clusters that can optimize performance and expenses according to specific service-level objectives. This extension of Kafka's capabilities not only enhances its economic viability but also increases its architectural flexibility, making it a versatile platform suited to a broader range of applications.

Aiven recently won the Top Tech Award in the “Transformation and Cloud” category from L’Informaticien, a prestigious IT publication in France, recognizing its innovative contributions to digital transformation and cloud technology. The award was largely attributed to Aiven's collaboration with Back Market, a leading marketplace for refurbished electronics, where Aiven facilitated a seamless migration of Back Market's real-time data infrastructure to Google Cloud. This migration improved data processing speed and reliability, reduced infrastructure costs, and minimized environmental impact. Aiven's approach emphasizes making complex cloud infrastructure manageable and sustainable, allowing clients to focus on customer value. This recognition highlights Aiven's commitment to delivering effective cloud solutions and sets the stage for further innovations in the industry. The official awards ceremony is set to take place in Paris, celebrating the accomplishments of various digital leaders.

Aiven has reached a significant milestone by surpassing $100 million in annual recurring revenue, a testament to its commitment to open source and developer-focused data infrastructure solutions. The company offers managed platforms for essential open source technologies like PostgreSQL, Apache Kafka, OpenSearch, and ClickHouse, providing developers with reliable, scalable, and secure data management without infrastructure burdens. Aiven's success is attributed to the trust and innovation of its customers, including notable names like WalkMe, OVHcloud, La Redoute, Priceline, and Sophos, who benefit from Aiven's streamlined services. The company attributes its growth to its dedicated global team, supportive partners, and the foundational vision of its founders and investors. Looking ahead, Aiven aims to continue expanding its offerings and investing in AI-powered features, with plans to revolutionize Apache Kafka through its innovative Diskless architecture, furthering its mission to empower developers and enhance data infrastructure in the cloud era.