April 2016 Summaries

The first global Kafka Summit, sponsored by Confluent, has taken place at the Hilton in San Francisco. With over 600 attendees from five countries speaking 27 languages, and support from 12 sponsors, the event showcased the growth of Apache Kafka technology across various industries. A hackathon was also held with more than 100 participants, resulting in new connectors for the Kafka community. The summit featured discussions on topics such as cloud services and data management, emphasizing their importance in improving government agency efficiency.

Apache Kafka's optimal performance and reliability are significantly influenced by its underlying infrastructure, making careful selection of storage options crucial, especially when deploying on AWS Cloud. Each Kafka cluster consists of brokers that manage data in topics, ensuring data persistence and fault tolerance through replication and rebalancing. EBS volumes are recommended for Kafka on AWS due to their consistent I/O performance and flexibility, allowing for quick reallocation in case of instance failure and ensuring clean shutdowns without data loss. The newer st1 and sc1 EBS offerings are particularly promising for Kafka's sequential I/O workloads, offering enhanced performance and reliability at reduced costs. While storage is a primary focus, other infrastructure components like CPU, memory, and networking are also vital for a Kafka cluster's capabilities, and future discussions will delve into these aspects. Additionally, the announcement of Apache Kafka 3.8.0 introduces various new features and improvements, highlighting ongoing advancements in the platform.

Confluent is hosting a Stream Data Hackathon on April 25th in San Francisco, focused on building real-time data pipelines using Apache Kafka's new features, Kafka Connect for data integration and Kafka Streams for distributed, fault tolerant stream processing. The event aims to help the community build connectors and learn how to build Kafka Streams applications. Participants can win prizes such as lunch with Kafka co-creator Jay Kreps or gift cards. Registration is open to anyone interested in participating.

The Apache Kafka community was busy last month with the release of a technical preview for Kafka Streams and a vote on a release plan for Kafka 0.10.0. The discussion around key proposals accelerated to meet the deadline, two release candidates were rolled out, but ultimately, the release was put on hold to incorporate additional changes. This pause allowed for further refinement and improvement of the upcoming release, which is expected to include several new features and updates. Some notable changes include rack-awareness in Kafka, improvements to the Connect REST API, and support for standard SASL mechanisms. The community is also actively discussing various KIPs, including proposals for administrative actions, protocol improvements, and time-based indexes. These developments aim to enhance the overall performance and functionality of Apache Kafka.

In this article, Michal Haris and Neha Narkhede explore the integration of Apache Kafka Connect and Kafka Streams to build a real-time stream processing application. They demonstrate how to use Kafka Connect to ingest data from Wikipedia IRC channels into a partitioned topic, which is then processed by Kafka Streams to extract and parse the raw IRC messages into WikipediaMessage objects. The application also uses KTable to continuously print updates on usage analytics. The authors highlight the benefits of using Kafka Connect and Kafka Streams, including simplicity, scalability, and fault tolerance, as well as the ability to run instances in parallel without sacrificing performance. They also discuss potential improvements to the integration between Kafka Connect and Kafka Streams, such as a first-class integration that would allow connectors to map directly to KStreams.