June 2019 Summaries

Confluent's clients for Apache Kafka® have recently released version 1.0, which includes significant improvements in the .NET client. The new API introduces strongly typed configuration classes and builder classes to provide more flexibility during client creation. Additionally, serialization APIs have been designed with flexibility in mind, allowing for advanced use cases such as integration with Confluent Schema Registry. A new general-purpose Message class has also been introduced, encapsulating the contents of a message. These improvements make the Confluent .NET client one of the more polished APIs for Kafka and pave the way for similar enhancements in clients for other languages.

The text discusses the integration of domain-driven design (DDD) and microservices with Apache Kafka, emphasizing Kafka's role as a leading event streaming platform that decouples microservices to create agile and flexible architectures. It highlights how DDD organizes complex business domains into bounded contexts, ensuring each microservice owns its domain logic, which is essential in preventing software fragility. Kafka's features, such as Kafka Streams, ksqlDB, and Kafka Connect, support this by providing powerful, decentralized event processing capabilities without centralizing business logic, thus avoiding traditional middleware pitfalls. The text also explains the importance of anti-corruption layers in maintaining the integrity of domain models across different contexts. Lastly, it touches on Kafka's new 3.8.0 release, which boasts various enhancements and invites users to explore further by downloading the Confluent Platform.

Apache Kafka 2.3 has been released, bringing several new features and improvements to the popular open-source event-driven platform. One of the key features is the addition of metrics that show partitions with exactly the minimum number of in-sync replicas, allowing users to monitor under-replicated partitions. Additionally, a new command-line flag allows users to easily see which topics have fewer than the minimum number of in-sync replicas. Other notable improvements include prioritizing existing connections over new ones, improving log recovery times, and adding in-memory implementations for the Kafka Streams window store and session store. The release also includes updates to Kafka Connect, including a more efficient rebalancing process and improved logging. Overall, these changes aim to improve the reliability, performance, and scalability of Apache Kafka 2.3.

The text discusses how Apache Kafka can be used to build a scalable and resilient search infrastructure by connecting different source applications, databases, and search engines. It introduces three approaches that use Apache Kafka to help scale search solutions, handling an increasing number of documents or integrating various sources of information. The article also covers the use of Kafka Connect for connecting systems across Kafka, as well as the JDBC connector for moving data while adapting it to the requirements of a search product. It emphasizes the importance of schema evolution and selecting the right mode of operation when using connectors. Finally, it mentions the Elasticsearch connector for pulling data from Kafka into Elasticsearch and highlights some considerations related to mapping inference and retries.

The field of astronomy is rapidly evolving into a data-intensive science with real-time needs. Apache Kafka® has proven to be an effective tool for managing and distributing large volumes of astronomical data, particularly in the context of transient object detection and analysis. The Zwicky Transient Facility (ZTF), which detects about 1 million changing objects per night, utilizes Kafka as its alert distribution system due to its reliability, ability to handle high-speed data streams, and support for structured data formats like Avro. This technology allows astronomers to collect, process, characterize, and distribute data on transient objects in near real time, enabling rapid scientific discovery and collaboration.

The text discusses the use of Confluent Cloud, a fully managed event cloud-native streaming service that extends the value of Apache Kafka®. It focuses on the specific Helm Chart configuration needed to connect the Confluent Platform components like the Confluent Schema Registry, Kafka Connect, KSQL, and Confluent REST Proxy to your Kafka cluster running in Confluent Cloud. The requirements for this are access to a Kubernetes cluster with version 1.9.2+ and Helm/Tiller 2.8.2+ installed. It also provides an example of updating the values.yaml file for KSQL to enable a connection to Confluent Cloud.

In part 4 of the Spring for Apache Kafka Deep Dive blog series, the focus is on building event streaming applications using Spring Cloud Data Flow, Apache Kafka, and Spring Cloud Stream. The article explains how to create and deploy event streaming pipelines that utilize Kafka topics as named destinations, allowing developers to produce and consume data to and from these topics with ease. It delves into the use of Stream DSL syntax for defining streams, illustrating how to handle user/click events and apply filtering, business logic, and storage in an RDBMS. The blog also covers advanced topics like fan-in/fan-out, partitioning for content-based routing, and function composition to dynamically attach business logic to streaming applications. Developers can employ the Spring Cloud Data Flow shell to register, deploy, update, and manage event streaming applications, with support for continuous deployment and version control. The series emphasizes the versatility and integration capabilities of Spring Cloud Data Flow in building and managing scalable event streaming solutions on Apache Kafka, while also highlighting new features in Apache Kafka 3.8.0.

TokenAnalyst is building the core infrastructure for integrating, cleaning, and analyzing blockchain data using Apache Kafka® as its central data hub. They offer historical and low-latency data streams of on-chain data across multiple blockchains. The Confluent Schema Registry helps manage multiple semantically enriched data models for the same underlying data. On the event streaming side, they provide reliable low-latency data streams for financial applications based on Kafka Streams. They also use KSQL to experiment with raw or lifted streams and deploy new machine learning models without writing Java code. The importance of on-chain data is expected to grow, prompting potential on- and off-chain data joins. TokenAnalyst leverages the Confluent Platform for building resilient data pipelines.