August 2018 Summaries

ksqlDB is an event streaming database that enables real-time data processing against Apache Kafka, providing SQL-like semantics for reading, writing, and processing streaming data in real-time at scale. To extend its functionality, ksqlDB supports creating user-defined scalar functions (UDFs) and user-defined aggregate functions (UDAFs), which can be built using Java classes with additional dependencies if needed. These custom functions allow users to add new capabilities to their ksqlDB statements without modifying the existing syntax. Building a UDF is straightforward, requiring only one Java class, minimal wrapper code, and deployment on the ksqlDB server or cluster. The example demonstrates how to create a powerful UDF using an autoencoder neural network for real-time anomaly detection in car sensor data, showcasing its potential for continuous processing of large event streams. With this capability, users can leverage custom functions without requiring extensive programming knowledge, making it accessible to both data engineers and scientists.

Apache Kafka and its related technologies are at the forefront of a paradigm shift in building modern data-centric applications, with event-driven architectures powered by Apache Kafka becoming increasingly prevalent across various industries. This shift is driven by businesses' increased digitization, the need for real-time decision-making, and advancements in stream processing technology. Stream processing views data as continuous flows to be processed continuously, enabling responsive, flexible, decoupled, and real-time applications. Kafka Streams is a stream processing library native to Apache Kafka that allows developers to build event-driven applications in Java to process data in Apache Kafka topics. It provides a versioned protocol for correct, distributed, fault-tolerant, stateful stream processing, making it an enabler of modern stream processing. The emergence of event-driven architectures powered by Apache Kafka is expected to have a significant impact on how companies utilize data, comparable to the impact of relational databases.

The text discusses the release of open-source Helm Chart deployment templates for Confluent Platform components on GitHub, enabling developers to quickly provision Apache Kafka, ZooKeeper, Schema Registry, REST Proxy, and Kafka Connect on Kubernetes. It also mentions a white paper providing best practices for deploying Apache Kafka on Kubernetes, which will be updated as the ecosystem evolves. The end goal is to make streaming data ubiquitous by combining Kubernetes' ability to run applications anywhere with Kafka's instantaneous accessibility of data anywhere.

The text discusses the benefits of using Confluent Schema Registry, which is included in the Confluent Platform, to achieve strong decoupling of systems integrated via Apache Kafka. It explains how schema evolution rules can reduce coupling between producers and consumers, making it easier for teams to be more agile and create robust applications that are resilient to change. The text also covers the use of Avro serialization format with Confluent Schema Registry, highlighting its flexible and well-defined rules around schema evolution. Additionally, it provides a practical example of how to use Avro and Schema Registry from C#, demonstrating how to produce and consume Avro serialized data using .NET applications.

The text highlights the transformative impact of streaming platforms on data management, emphasizing the shift from static databases to dynamic, event-driven systems that enable real-time data manipulation and application activation. It announces the integration of the Seattle-based Distributed Masonry team, known for their Onyx Platform and Pyrostore innovations, into Confluent to enhance the cloud-native capabilities of Confluent Cloud. This collaboration underscores the evolving landscape of cloud-native technologies, which are driven by different dynamics compared to traditional on-premise systems, and hints at future innovations in the sector. Additionally, Confluent's recognition as Microsoft's 2024 OSS on Azure Global Partner of the Year is celebrated, marking its commitment to open-source solutions on Microsoft Azure. The narrative concludes by framing data streaming as a pivotal component of modern business infrastructure, signifying a major shift in its role and importance.