July 2019 Summaries

Confluent Platform 5.3 is now available, offering several key features to simplify Kafka operations, automation of deployment and lifecycle operations on Kubernetes, and enhanced security through the introduction of Role-Based Access Control (RBAC). The platform aims to provide a complete set of software interfaces, including a redesigned user interface for Confluent Control Center, a new CLI, and automated deployments using Confluent Operator and Ansible Playbooks. Additionally, Confluent Platform 5.3 introduces features such as Secret Protection, which encrypts secrets within configuration files, and support for cloud-native operational models on Kubernetes. The platform is built on the latest version of Apache Kafka (2.3) and provides a comprehensive security strategy to ensure access to resources in a way that is simple enough to reason about yet flexible enough to implement organization's security policies with precision.

The text discusses using Jaeger tracing in Apache Kafka® projects to trace how data flows through a distributed system. It provides an example project that uses Kafka Streams, a Kafka consumer/producer, and Spring Kafka consumer/producer. The author then explains how to make the Jaeger tracing solution more robust by using Kafka to send spans to Jaeger instead of directly communicating with it. This approach ensures fault tolerance, storage, and replayability for the distributed system. Additionally, the text mentions that Confluent Cloud can be used as a fully managed event streaming platform powered by Apache Kafka.

The Kafka Summit Program Committee has released the schedule for its San Francisco event, featuring two days of talks on various topics related to Apache Kafka®. With four tracks available, attendees can expect a diverse range of presentations catering to different interests and roles within the community. Some highlights include discussions about running Kafka at scale, ETL architectures, event-driven microservices, and internals. The event will take place on September 17-18 in Austin, Texas, with opportunities for networking and knowledge sharing among data streaming professionals.

MongoDB and Apache Kafka serve as foundational components in many modern data architectures, with Kafka Connect facilitating seamless integration between Kafka and external systems like MongoDB. The official MongoDB Connector for Apache Kafka, now in beta and verified by Confluent, allows MongoDB to function as both a source and a sink for Kafka, enabling data to flow bidirectionally. This integration supports various event-driven use cases, such as automatically replenishing inventory in an e-commerce setting when stock levels fall below a threshold. The process involves setting up and configuring the MongoDB Connector through Confluent Hub, utilizing change streams to trigger Kafka events, and employing tools like kafkacat to test scenarios. The connector, derived from Hans-Peter Grahsl's original sink connector, now integrates source capabilities and is maintained by MongoDB engineers, highlighting its suitability for event-driven architectures and microservices. Additionally, the document mentions the release of Apache Kafka 3.8.0, which includes several enhancements, and touches on a playful debate regarding coding style preferences, illustrating Kafka's versatility in various applications.

The text discusses how to build a burglar alarm system using network-connected cameras, Apache Kafka®, Kafka Streams, and TensorFlow. It provides guidance on selecting the right camera, setting up the cameras for surveillance, and processing the images from the cameras. The author also explains how to create an automated burglar alarm by building a TensorFlow model in the Kafka Streams pipeline to detect burglars. Additionally, it covers configuring a sink connector that will send burglar alert messages and using a smart lock system for filtering out unwanted alerts. The complete source code of the demo application can be found on GitHub.

A new KSQL course has been created by Stéphane Maarek and a developer to help inspire and guide developers in embracing event streaming technology. The hands-on course teaches students how to process data in Apache Kafka using an interactive SQL interface, covering streams and tables, advanced KSQL areas, production deployment lessons, geospatial extensions, user-defined functions, and more. The course consists of 33 lectures under 10 minutes each, aiming to provide a quick yet comprehensive introduction to KSQL for both beginners and experienced developers.

In this three-part series, we discussed building an event streaming architecture using Apache Kafka, KSQL, and Kafka Streams for a customer project. We also explored how Gradle helped address the challenges faced during development, building, and deploying the KSQL portion of our application. Additionally, we delved into using Gradle to build and deploy KSQL user-defined functions (UDFs) and Kafka Streams microservices. In part 3, we focused on using Gradle to build and deploy UDFs and Kafka Streams microservices. We utilized the Spock framework for writing test specifications and employed Groovy instead of Java to write our UDFs due to its ability to handle data types more efficiently. Furthermore, we demonstrated how to create distribution artifacts complete with start scripts that can be packaged up as part of the build process using the Application plugin. Throughout this series, we highlighted how Gradle can be used to build all things Kafka with both built-in functionality and plugins contributed by the community. We also introduced the gradle-analytics plugin, which captures information about what's going on inside of Gradle and sends that activity to various cloud services like AWS Kinesis, Amazon S3, Cloud Pub/Sub, or Google Cloud Storage.

This blog post explores sentiment analysis using Apache Kafka and Google Cloud's Natural Language API for the FIFA Women’s World Cup. The author uses pre-built connectors available in Confluent Hub to source data from RSS feeds of sport-related newspapers and Twitter feeds of the FIFA Women’s World Cup. They then use KSQL to apply necessary transformations and analytics, Google's Natural Language API for sentiment scoring, Google BigQuery for data storage, and Google Data Studio for visual analytics. The author demonstrates how to ingest Twitter and RSS event streams into topics, shape them with KSQL, implement Natural Language API calls in a UDF, push the dataset into Google BigQuery, and visually analyse the data in Google Data Studio. They also provide an example of sentiment analysis for a specific match between Norway and England.

The text discusses Kafka listener configuration and its importance in ensuring smooth functioning of Apache Kafka, a distributed streaming platform. It explains that clients need to be correctly connected to Kafka brokers, which can have multiple listeners. Misconfigurations can lead to problems such as connection failures or metadata resolution errors. The post provides solutions for Docker and AWS configurations, emphasizing the importance of setting advertised.listeners properly. It also mentions Confluent Cloud as an alternative solution where someone else handles Kafka configuration and maintenance.