April 2019 Summaries

The Kafka Streams DSL introduces optimization techniques to improve efficiency in stream processing applications. The framework generates a logical plan before compiling it into a physical plan, allowing for optimizations such as reducing unnecessary repartition topics and state store footprints. The `StreamsConfig.TOPOLOGY_OPTIMIZATION` configuration can be set to enable topology optimization, which is enabled by default in newer versions of Kafka Streams. This feature aims to improve performance and reduce resource usage in stream processing applications.

Worldwide food processing machinery manufacturer BAADER is using Apache Kafka® and Confluent Cloud to capture Internet of Things (IoT) event data from farm to fork, increasing efficiency in the food value chain. By capturing events all along the food value chain, processing those events in Confluent Cloud, and sharing insights with interested parties throughout the chain, BAADER is not only increasing the value it provides but also making food production more intelligent and efficient. The company leverages KSQL and IoT data from cameras in their processing facilities to calculate machine efficiency and alerts factory managers about potential drops in efficiency. They also collect delivery truck GPS data and integrate with weather data to help companies understand how traveling conditions affect product quality and animal welfare. Ultimately, BAADER wants consumers to be able to give feedback on quality via a mobile app, which will allow the company to analyze the data in Confluent Cloud and determine where participants in the food value chain can make changes to improve product quality and the consumer experience.

The Test Machine is a library developed by Funding Circle to test the integration of multiple systems in their event-driven architecture, which relies on Apache Kafka and other technologies like Ruby on Rails, Samza, and Kafka Streams. The system was experiencing flaky and slow tests due to subtle errors in the test code, but after analyzing the fixes, they realized that many tests matched a simple pattern. To address this issue, they created a pure data interface using Clojure and implemented the Test Machine to decouple test authors from the mechanics of writing test data to Kafka. The library provides a way to define tests as sequences of commands consisting of writes and watches, which can be executed against various targets such as mock topologies or real Kafka clusters. This approach enables full-stack testing with I/O and allows for faster test execution times. The Test Machine is now part of the Jackdaw Clojure library used by Funding Circle to develop event streaming applications on the Confluent Platform.

The text discusses the evolution and current state of Apache Kafka's client APIs, which initially supported only Scala and Java but have since expanded to accommodate a wide range of programming languages, allowing developers to choose the language best suited to their needs. Confluent plays a significant role in developing and enhancing these Kafka clients, ensuring their robustness, scalability, and interoperability across different Kafka versions. The open source community has contributed to this ecosystem by providing additional language support through extensions built on the C client library, librdkafka. Confluent offers examples and resources on GitHub to help developers get started with Kafka, and the fully managed service Confluent Cloud provides an accessible option for deploying event streaming applications. Additionally, the release of Apache Kafka 3.8.0 introduces new features and improvements, and a playful mention of the "tabs vs. spaces" debate is used to demonstrate Kafka's capabilities with a JavaScript client example.

Confluent Cloud and Apache Kafka are at the center of the Internet of Things (IoT) transformation in various industries, including food processing and retail. Confluent Platform enables IoT-based disruption and innovation by providing a scalable, real-time pipeline for collecting, transporting, and storing IoT events globally. Companies like BAADER and Mojix are leveraging Confluent Cloud to build data-driven solutions that capture insights from IoT devices, improving operations such as inventory management, animal welfare, and customer experience. By capturing and analyzing vast amounts of IoT data, these companies can identify opportunities for improvement and drive business growth. The use of Confluent Cloud provides a resilient, scalable, and event streaming service that enables developers to address various IoT-related challenges, including complex event processing, transformations, and connectors.

The latest release of Confluent Platform introduces new features that enable the building of contextual event-driven applications. Among these is the enhanced integration between Control Center and Confluent Schema Registry, which allows for centralized schema management and evolution while maintaining compatibility. This enables client applications to register and retrieve globally unique schema IDs, inspect topic data, view schema changes, and modify compatibility policies. The main value of Schema Registry is enabling schema evolution as applications and data develop over time, ensuring that all applications relying on old and new versions of a schema remain compatible.

The recent Kafka Summit in New York showcased the latest developments and innovations in the world of data streaming, featuring keynotes from industry leaders like Jay Kreps of Confluent, who emphasized the importance of integrating events and state in system infrastructure, and James Watters of Pivotal, who highlighted Kafka's role in transforming monolithic architectures into dynamic microservices ecosystems. The summit, planned to be smaller than the previous San Francisco event, included the announcement of Confluent Platform 5.2, offering new features for developers. Attendees enjoyed a variety of highly-rated sessions, reinforcing the vibrant community surrounding Apache Kafka. Anticipation builds for upcoming events in London and Austin, Texas, where discussions will continue on topics related to Apache Kafka and Apache Flink, while Confluent's collaboration with AWS aims to enhance cloud service management for improved efficiency and customer experience in government agencies.

The text discusses the development and functionality of the Kafka Connect S3 connector, released in March 2017 as part of the Confluent Platform, which allows users to stream data from Apache Kafka to Amazon S3, a crucial component of many AWS architectures. The connector was designed from scratch to meet user needs for reliability and exactly-once semantics, overcoming limitations of existing solutions by leveraging multipart uploads and treating Kafka as the sole source of truth to ensure efficient and robust data handling. The document highlights the ease of use and reliability of the connector, which has since been used to upload over 75 PB of data, and provides a detailed explanation of its setup and operation, including authentication, configuration, and the partitioning of records using timestamps. The text also invites users to contribute to the development of the S3 connector and announces the release of Apache Kafka 3.8.0, which includes new features and improvements.

The Confluent Platform provides an integrated event streaming architecture that enables enterprises to run modern data systems and services across multiple cloud providers, private clouds, and on-prem deployments. The platform's Confluent Replicator enables frictionless data replication between sites, allowing data to stay in sync in near real-time between core business applications regardless of their location. With Confluent Control Center, operators can manage multi-datacenter Apache Kafka deployments, view topic data and schemas, run ksqlDB queries, and monitor the performance of client applications. Control Center also provides insights into how client applications are performing and allows users to set up data replication between clusters, monitor Replicator's consumer lag, and derive topic-to-client mappings. The platform is designed to ensure that mission-critical, multi-datacenter Apache Kafka deployments have data replicated and stays in sync in near real-time, making it an essential tool for enterprises with complex data architectures.

The announcement highlights a new partnership between Confluent and Google Cloud, making Confluent Cloud's managed Apache Kafka service available as a native offering on Google Cloud Platform (GCP). This integration allows users to leverage Confluent Cloud's capabilities with familiar Google tools, offering a seamless sign-up experience, integrated billing, and support from Google Cloud. The move underscores the evolving landscape of cloud-native data systems, where open-source platforms are transforming to offer the elasticity and flexible pricing of proprietary cloud services. Confluent, as a cloud-native pioneer with deep expertise in Apache Kafka, aims to provide these benefits while maintaining the rich ecosystems and portability of open-source software. The collaboration with Google Cloud is part of a broader trend, as other open-source companies like Elastic, MongoDB, and Neo4j also integrate their services with GCP, showcasing a model of cooperation that reflects the growing openness and ecosystem expansion of cloud platforms. Meanwhile, Confluent Platform 7.7 is introduced, offering enhanced security features and new capabilities like OAuth support and integration with Apache Flink and Amazon OpenSearch.

The event-driven architecture of microservices is based on events that drive actions in each service. In this paradigm, services act on a common stream of events, which can be processed using the Kafka Streams library to build streams from data in a Kafka topic and take action on them. To handle specific requirements, such as processing only relevant events or dynamically routing output topics, branching or dynamic routing capabilities are used. These capabilities allow microservices to subscribe to separate streams or produce data into new Kafka topics based on certain conditions or derived from event fields. The application assumes advance knowledge of the output topic names but can also use a TopicNameExtractor to dynamically determine the name of the Kafka topic to send records, enabling dynamic routing. This feature allows for flexible and efficient processing of events in various domains, such as finance, retail, and IoT.

KSQL 5.2 introduces new features and enhancements that expand its SQL-based streaming capabilities, allowing users to perform complex data transformations and analyses in real-time. Notable additions include support for the CASE function, which enables flexible data manipulation and cleaning, and the ability to log processing errors to Kafka topics for easier inspection and querying. The release also improves user experience with features like the PRINT command for displaying Kafka topic contents and URL parsing functions. These updates enhance KSQL's maturity and functionality, making it a more powerful tool for building streaming applications and facilitating analytical processes. The blog encourages exploration of the new capabilities in Confluent Platform 5.2 and hints at further advancements with its successor, ksqlDB.

Confluent Platform 5.2 is now available, offering significant enhancements in three key dimensions: building contextual event-driven applications, managing larger and more complex Kafka deployments, and enabling hybrid event streaming leveraging Confluent Cloud. The platform introduces a new Developer License, allowing developers to run all commercial features for free on single-broker Kafka clusters without time constraints. This license provides unrestricted access to Control Center, KSQL, and other commercial features, making it easier for developers to experiment freely and solve problems. Additionally, the release includes major enhancements to Control Center, such as improved schema management, dynamic broker configuration changes, and support for multiple Kafka Connect and KSQL clusters. Confluent Platform 5.2 is built on Apache Kafka 2.2.0, which includes new features, performance improvements, and bug fixes. The platform also enables hybrid event streaming with the ability to replicate Schema Registry to Confluent Cloud, making it easier to manage larger and more complex deployments. With these enhancements, developers can build contextual event-driven applications faster and more efficiently, while managing their Kafka environments at scale.