March 2022 Summaries

Companies are increasingly adopting Karapace, an open-source alternative to Confluent Schema Registry and REST API for Apache Kafka®, started by Aiven. The software handles the distribution and evolution of schemas outside a Kafka cluster. Aiven created Karapace in response to licensing changes in Confluent Schema Registry back in 2018, aiming to provide an open-source alternative for all Apache Kafka users. Aiven is committed to continuing the development of Karapace and ensuring compatibility between Confluent Schema Registry and Karapace. The collaboration with Instaclustr has further strengthened the future of Karapace by adding support for the Google Protocol Buffer (Protobuf) format in the Karapace schema registry, which allows cross-language support for code generation.

Apache Kafka is a distributed event streaming platform that allows applications to produce, consume, and process records asynchronously. Key components include brokers (servers), consumers (applications reading data), consumer groups (coordination for multiple instances of an application), producers (applications writing data), topics (logical channels for organizing messages), partitions (data split across servers), replication (data preservation across multiple servers), and event-driven architecture. Apache Kafka supports both pub/sub and queueing messaging architectures, with features like message ordering, watermark tracking, and state management. It is designed to handle large volumes of data and provides high throughput, low latency, and fault tolerance.

Apache Kafka is a versatile platform that helps businesses address various use cases. Companies such as Comcast, Paf, OVO Energy, Alef Education, GoTo Financial, Mirakl, JobCloud, Norauto, and NetSpyGlass have successfully implemented Aiven for Apache Kafka to streamline their operations. These companies leverage Kafka's low latency, scalability, and ability to bridge the gap between data collection and analytics. By integrating Kafka with other technologies like OpenSearch, PostgreSQL, Redis, InfluxDB, and Grafana, these businesses have achieved modern, extensible, and scalable solutions tailored to their specific needs.

The article discusses the differences between two popular DevOps tools - Kubernetes and Terraform, both of which are open-source projects. While Kubernetes is an orchestration tool for managing containers, Terraform lets users define their infrastructure as code. Both tools allow declarative definition of systems and offer modularity and reusability. They can be used together to create/manage infrastructure and manage containers that house applications. The article also explains how these tools can be used to provision data infrastructure on Aiven, a managed cloud services platform. It concludes by suggesting that the choice between using Terraform or Kubernetes depends on factors such as existing familiarity with the tool, whether other infrastructure resources are being managed using it, and whether there is a need for a single control plane to manage both application and data resources.

On March 7th, 2022, a vulnerability called "Dirty Pipe" (CVE-2022-0847) was discovered, which allows unprivileged local users to write to pages in the page cache backed by read-only files. This could potentially increase their access and enhance their privileges within the system. Aiven's CISO has outlined mitigating actions taken against this vulnerability, including an optional maintenance update for all customers that will be made mandatory over the next 30 days. The Aiven platform does not allow direct interaction with the underlying operating system, and its architecture prevents cross-tenant impact from such vulnerabilities. Additionally, internal monitoring has been extended to help identify any exploitation attempts.

Apache Kafka® is a powerful open-source distributed event streaming platform, but it lacks an opinion about the structure of data. To address this issue, Karapace provides an open source HTTP API interface and schema registry for Apache Kafka®. Key features include a schema registry that allows producers to validate their payload against expected schemas before producing records on topics, and RESTful API endpoints for Apache Kafka, which enable working directly with data and metadata over HTTP. Karapace is an open source Python-based project released under the Apache 2.0 license, making it easy for developers to contribute and enhance its functionality.

Apache Flink 1.15.0 introduces new JSON SQL functions, allowing more powerful data processing. The use case focuses on IoT sensors that emit sensor reading messages in JSON format with possible nested JSON fields. With the help of these new functions, users can parse and build nested JSON data structures using only SQL statements. This feature will be available in version 1.15.0 but can already be explored by cloning Apache Flink's source code and building it yourself or waiting for the official release.

Apache Kafka® and Apache Flink® now support real-time data processing and streaming analytics without the need to write code, using simple SQL statements in Flink. This open source tool allows for collaboration while maintaining rigorous data governance. With Flink jobs, users can filter, window, or enrich data according to their needs, making it an ideal platform for running data experiments on. Flink also offers a highly available platform for deploying data pipelines and is configured with cluster restart strategy on fault tolerance. It maintains state in the Flink job, simplifying complex SQL joins and data windowing.

The article discusses the process of migrating an Elasticsearch client to OpenSearch® in Python, Java, and Node.js. It explains that when migrating databases, it is essential to consider migrating clients as well. The author provides a brief overview of database migration from Elasticsearch to OpenSearch and then delves into client migration for each language. They recommend upgrading or downgrading the Elasticsearch version before migrating to the compatible OpenSearch version v1.0.0, which can help correct API incompatibilities during the process. The article also mentions that Aiven offers an easy way to perform an upgrade from its console with a click of a button.