March 2025 Summaries

The author attended pgDay Paris 2025, a one-day PostgreSQL conference in Paris, which was their first PostgreSQL conference. The event had a strong sense of community and high-quality talks on various topics such as full-text search, Kafka & Debezium integrations, and more. Notable talks included "Tuning Community Parameters: Hosting PostgreSQL Events that Perform", "Full-Text-Search Explained from A to Z with French Food", and "Why PostgreSQL people should really care about Kafka and Debezium?". The author also participated in lightning talks and social events, including a karaoke session. They expressed their enthusiasm for the event and look forward to attending future conferences like PGDay UK and PGDay Chicago.

Apache Kafka Tiered Storage is a feature that allows data to be stored in a secondary cold storage system, reducing the load on the primary storage. This is achieved through KIP-405, which introduces an external store where most of the cluster's data is stored. The plugin, developed by Aiven, supports all three major cloud object stores and has been battle-tested in their Kafka fleet. Tiered Storage works by asynchronously tiering data to the external store, keeping both a local and remote copy for a subset of the data. Reads can come from either the external store or the local data, with preference given to the local one. The plugin uses chunk-based uploads to store data in the external store, which allows for efficient caching and prefetching mechanisms. Metadata is stored in an internal topic named `remote_log_metadata`, which is consumed by each broker to build up the latest remote segment state in memory. This feature allows users to build additional custom functionality on top of Kafka, making it a simpler and more cost-effective solution for large-scale data storage.

Aiven has made significant progress in its sustainability journey over the past year, from refining policies to improving data collection and aligning with evolving regulatory expectations. The company conducted a double materiality assessment, CSRD gap analysis, emissions data remediation, introduced a supplier code of conduct, published its first UK Modern Slavery Statement, and drafted key ESG policies and KPIs to measure progress in different areas. Building on this momentum, Aiven aims to publish its first sustainability report, refine its policies and data collection processes for CSRD readiness, expand internal sustainability training, develop its carbon footprint tool to provide actionable data and drive emissions reductions, and make continuous improvements to build a more responsible and resilient company.

Aiven offers a robust ecosystem of tools and integrations to support both strategies of storing first and then analyzing, or analyzing first then storing, for managing logging and metrics data. Aiven provides managed data services such as OpenSearch, Apache Kafka, and ClickHouse, which can help businesses store and analyze their log and metric data effectively. Additionally, Aiven offers monitoring solutions like Aiven for Metrics (backed by Thanos) and Aiven for Grafana, which integrate with external providers like Amazon CloudWatch, Google Cloud logging, and Datadog. The choice between storing first and then analyzing, or analyzing first then storing, depends on the specific needs and priorities of the organization, with the former approach offering greater reliability suitable for most use cases, and the latter excelling in real-time observability and time-sensitive applications. Aiven simplifies the complexities of log and metric management, allowing businesses to focus on deriving value from their data.