November 2021 Summaries

Apache Kafka version 3.0 was announced on September 21, 2021, and Aiven already supports it. This new major version comes with enhancements such as grouping multiple consumer group offset requests, stronger default delivery guarantees by the producer, identifying the liveliness of a topic/partition, more granular restarting of connectors, and the introduction of Karapace schema registry for Apache Kafka 3.0. Deprecations include MirrorMaker v1, Java 8 and Scala 2.12 programming language versions, message formats v0 and v1, and IDEMPOTENT_WRITE. Users can launch a brand new cluster in the Aiven Console or conduct a no-downtime upgrade to the latest version.

Panu Kalliokoski, Senior Software Engineer at Aiven, shares his experience of convincing management to adopt Aiven as a solution for running data-oriented background services such as databases in a modern and automated way. He discusses the challenges faced when trying to get everyone on board with this approach, particularly within more established organizations where personal factors and power struggles can make it difficult to agree on how things should be done. Kalliokoski provides arguments for using Aiven over existing cloud solutions, addressing concerns about vendor lock-in, integration with existing infrastructure, and the cost of managing data services in-house.

Companies are increasingly setting up dedicated teams to handle tooling issues for developers, allowing them to focus on their core competence of product development. These teams, known as Developer Experience (DEX) or Cloud Operations (CloudOps) teams, manage tasks such as maintenance of local and cloud software tools, upkeep of libraries and SDKs, subscriptions to various services, oversight of supported technologies, best practices around workplace collaboration, creation of supporting materials, and working with corporate IT. By centralizing these functions, companies can achieve cost savings through increased bargaining power and ensure that all tools meet security and scalability requirements. Additionally, a CloudOps team helps optimize resource management, making it environmentally friendly by reducing unnecessary consumption of resources. Overall, having a dedicated team to handle tooling issues benefits everyone in the company, from developers to the organization as a whole.

The Aiven Cluster Startup Program offers fast-growing startups free credits worth up to US$100,000 to be used over 12 months for building modern data infrastructure in the cloud without management headaches. In addition to financial support, Aiven provides access to its experts and technical support, virtual events focused on building the best data infrastructure for companies, joint promotions and visibility on Aiven's channels for startups that scale up within six months, and opportunities to give back to the community. The program is open to eligible companies that are affiliated with an approved partner organization, less than five years old, at Series A funding level or below, and have a company website and associated email address.

PostgreSQL 14.0 has been released with several new features and improvements, including stored procedures returning data via OUT parameters, SQL-standard SEARCH and CYCLE options for common table expressions, subscripting for any useful notation data type, multiranges for range types, performance enhancements for parallel queries, B-tree index updates efficiency, more aggressive VACUUM operations, extended statistics on expressions, and pipelined multiple queries in libpq. Users can upgrade to PostgreSQL 14.0 to take advantage of these features, with Aiven users able to run an in-place upgrade for migration.

Aiven services now support AWS PrivateLink for secure network connectivity, adding to the previously supported VPC peering and AWS Transit Gateway options. AWS PrivateLink provides private connectivity between VPCs, Aiven services, and on-premise networks without exposing traffic to the public internet. This enhances security by reducing vulnerability to attacks and simplifies network management. To get started with AWS PrivateLink in Aiven, see the help article "Using AWS PrivateLink with Aiven services."

Telegraf is a popular open-source metric collection tool with over 200 plugins, but it can overwhelm downstream systems like InfluxDB or Prometheus when collecting metrics from thousands of Telegraf agents. M3, an open-source time series database developed by Uber, offers easy horizontal scalability and compatibility with standard industry interfaces for producing and consuming metrics. Aiven is one of the first cloud providers to offer M3 as a service, allowing users to retain their current Telegraf agents while benefiting from M3's scalability. By integrating Telegraf with M3 and using an Aiven for Grafana service for visualization, users can efficiently manage metrics collected by thousands of Telegraf agents.

Many Database Administrators (DBAs) resist moving from on-premise to managed database services, but there are several benefits to consider. Managed services simplify operations and reduce expenses by packaging processes and functions together for customers. DBaaS (Database as a Service) allows subscribers to outsource the hosting and management of their databases. With managed solutions, companies can manage with fewer staff members, allowing developers, data scientists, and DBAs to focus on tasks they find meaningful. Managed cloud solutions also eliminate some risks associated with in-house database management, as experts handle infrastructure concerns. Additionally, managed services often cost less than on-premise solutions when considering all expenses involved. Companies using Aiven's managed services can benefit from 24/7 technical support and focus on their core business objectives.

Apache Flink and Apache Kafka are often used together due to their complementary features, with Flink being a powerful tool for analytics and understanding the architecture of Kafka, while Kafka is performant and scalable. The combination of these two technologies can greatly enhance data streaming capabilities. However, challenges may arise when working with skewed data, which can be mitigated by adjusting metrics and reviewing partitioning mechanisms. While Flink has improved significantly over time, there are areas where it could still improve, such as operational support for SQL queries and savepoint compatibility between SQL jobs. Alternatives to Flink include kSQL, but Flink offers more flexibility due to its wide range of connectors. For those new to Flink, the most fun thing to try out is exploring the documentation and trying out the SQL client.

The aiven-laravel package has been released to make Laravel developers' lives easier by adding helpful functionality to their projects. This package makes it easy to reconfigure your database connection, allowing you to connect to any Aiven database and power off unused ones. It also supports use of the DATABASE_URL environment variable for a more convenient way to work with database connections. Additionally, the package allows users to power off databases when not in use to save costs. The Aiven-Laravel package is open source and available on GitHub.