March 2023 Summaries

Romaric Philogene from Qovery and Sébastien Blanc from Aiven demonstrate how their respective platforms, Aiven for PostgreSQL® and Qovery, can be used together to streamline app development and deployment on AWS. They explain the benefits of using these tools in combination, including the ability to create multiple environments and avoid staging starvation issues. The integration between Aiven and Qovery is shown through a sample Java application that uses Aiven for PostgreSQL® as its backend database.

Apache Kafka is known to preserve message ordering per topic/partition, but there can be exceptions due to network issues or multiple producers. A single producer scenario may result in unexpected series of messages if a message gets delayed or lost. To ensure proper delivery and ordering, acknowledgment mechanisms and retries should be set up. Idempotent producers can also help maintain the global ordering of messages. For scenarios with multiple producers, it is crucial to handle event time properly on the consumer side. Some tools like Kafka Streams, Kafka Connect, and Apache Flink allow defining which field to use as event time for reshuffling message order.

Platform teams are becoming increasingly important in delivering consistent experiences to developers and engineering teams, leading to higher productivity and velocity. The most popular platform for delivery is Backstage, a CNCF project donated by Spotify. However, the ecosystem is still immature with many gaps and integrations within Backstage. Developer portals like Backstage enable ownership and accountability among development teams, allowing them to be responsible for their own costs and operations. These portals also improve productivity through faster onboarding of developers.

Aiven has introduced a fully-managed Apache Flink® service that provides a SQL interface to define streaming and batch data transformation pipelines on top of various technologies like Apache Kafka®, PostgreSQL® or OpenSearch®. The company's journey with Apache Flink began when it needed a tool for unifying pipeline definition across different backend technologies. Since the first release of Aiven for Apache Flink beta, several improvements have been made to enhance the developer experience, including an abstraction layer on top of Apache Flink SQL, smoother pipeline definition, faster definition and reuse, tighter control on deployments, improved lifecycle management, and more. The future of Aiven for Apache Flink includes adding new integration and connection capabilities, integrating all upcoming Flink SQL features, and accommodating new use cases and needs from existing and new customers.