May 2025 Summaries

PostgreSQL, widely regarded as the world’s most advanced open-source relational database, continues to set the standard for performance, scalability, and flexibility in modern data infrastructure. The database has undergone significant advancements since the release of PostgreSQL 13 in 2020, introducing powerful features and enhancements that make it an even more compelling choice for developers, database administrators, and IT leaders alike. Staying up-to-date with the latest PostgreSQL versions is a best practice and a strategic advantage for any organization relying on data-driven applications, as each new version introduces optimizations that improve query execution times, resource utilization, and overall database performance. Key features introduced since PostgreSQL 13 include incremental sorting, parallelized vacuuming of indexes, improved JSON support, enhanced security measures, community and ecosystem support, and logical replication enhancements. Upgrading to the latest PostgreSQL version can bring numerous benefits, but it requires careful planning and testing, with two main approaches to upgrading: in-place upgrade and copy upgrade. The latest version has introduced features like SQL MERGE command, LZ4 compression, and enhanced logical replication, providing developers with powerful tools to build efficient, scalable, and secure applications.

The PostgreSQL extensions ecosystem is a vital feature of the database, enabling it to adapt to modern data use cases such as AI, analytics, and document data stores. This flexibility keeps PostgreSQL viable for businesses and startups, making it easier to "outgrow" the database. The ecosystem allows developers to deploy extension code on a running PostgreSQL database with ease, but there are risks involved due to the open-source nature of the extensions. To mitigate these risks, using a managed service like Aiven can provide quality vetting and automatic backups. Some notable extensions include TimescaleDB for time-series data, PostGIS for geospatial objects, and pgvector for AI and ML applications. The PostgreSQL community is at a crossroads, balancing wide adoption with the need for reliable maintenance and management, and Aiven is eager to see how this plays out.

Diskless Kafka is a new feature that allows Apache Kafka to write directly to object storage, eliminating the need for local disk replicas and reducing costs by up to 80%. This feature, part of KIP-1150, enables users to choose between low-latency streams and ultra-cheap backfills for their topics. Diskless is designed to keep everything - including low-latency streams and cheap backfills - inside one logical log. The feature has already been tested in production by Aiven, a managed service provider, where it showed significant cost savings and reliability improvements. Diskless is being released as an open-source fork called Inkless, which allows users to test the feature on their own clusters before it ships upstream. With Diskless, Kafka's original vision of being a friction-free data mobility layer becomes a reality, with organizations able to keep adding topics without having to build custom solutions or incur additional costs. The feature is expected to accelerate Kafka's adoption and improve its network effect, as users can now dial storage economics per topic, reducing the need for tiered storage solutions.

Aiven represented at PGDay Chicago 2025, a conference that offered a range of talks on PostgreSQL database administration, performance optimization, and use cases. The event was great for anyone involved in Database Administration, including beginners, with a focus on student-friendly PGDataDay by Prairie Postgres. Key takeaways included the importance of using the right tools and extensions to achieve fast performance, managing tuple bloat and dead tuples, and leveraging PostgreSQL's flexibility as an open-source database. The conference also highlighted the potential of PostgreSQL as a versatile tool for various use cases, including PubSub patterns and hybrid search. Attendees were mostly professionals in the Chicago tech space, eager to share their experiences in the industry.

May 2, 2025 marked a significant event in the German Python community as PyConDE + PyDataDE took place, with Aiven attending and being represented by Celeste Horgan. The conference had an academic flavor due to its proximity to prestigious universities and research institutions. Dr. Guadalupe Canas-Herrera's keynote on mapping dark matter and dark energy using the Euclid telescope showcased the challenges of managing vast data quantities and the importance of deep learning and machine learning algorithms in this context. Aiven's involvement in open-source projects, such as the Euclid quick data release, was also highlighted. The conference featured talks on real-time analytics, AI/ML use cases, and Model Context Protocol, which emerged as an exciting new idea in the field. Aiven released its Aiven Model Context Protocol server for customers to work with large language models using their data stores. Community outreach programs sponsored by PyConDE demonstrated tangible financial benefits and brought talent into the field. The Aiven DevReal team will continue to support open-source communities and projects in the coming year, attending events like Devoxx UK, PGConf DE, PyCon US 2025, AWS Summit Hamburg, and Berlin Buzzwords.

The KIP-405 Tiered Storage in Kafka enables near-local performance with remote data by utilizing intelligent prefetching and caching. The plugin fetches data from the object store using a byte-ranged GET feature, which allows for efficient fetching of chunks. Caching is used to reduce round-trip times, and pre-fetching ensures that anticipated future reads are cached. The broker serves data from local disk first, and only one partition per fetch request is fetched from the remote store due to a limitation in KAFKA-14915. Deletes are handled by the new local retention settings, which delete data according to the local retention policy if it has successfully been tiered. Orphaned segments are cleaned up once their max leader epoch falls out of the leader epoch checkpoint file. The plugin fetches exactly one chunk per fetch request, and the offset index for the remote segment is fetched from the remote store and cached locally using a Caffeine W-TinyLFU cache.