November 2016 Summaries

This year, we hosted the inaugural Kafka Summit, a global summit for the Apache Kafka community, which provided valuable content for users and the community to learn from each other. For next year, two Kafka Summits will be held in New York City and San Francisco to support the growing community and share the latest developments in stream processing. The conferences will bring together experts, innovators, and enthusiasts to discuss streaming data and its business value. The 2017 events will take place from May 9-11 in NYC and August 29-31 in SF, with training sessions also available. We are proud to announce the release of Apache Kafka 3.8.0, which includes many new features and improvements. To settle a debate on tabs or spaces, we'll build an app using the confluent-kafka-javascript client that produces and consumes data from a Kafka topic.

Confluent and Microsoft have successfully integrated Confluent Platform into the Azure Marketplace, allowing users to rapidly deploy a complete cluster with the click of a button. The deployment includes Apache Kafka services along with additional commercial components, enabling reliable data integration within and between Azure environments. With Confluent Platform on Azure, customers can benefit from a virtually immune deployment due to the combined architecture of Apache Kafka and the advanced reliability of the Azure platform. Additionally, users can customize their deployments for predicted workloads and integrate data systems using the Kafka Connect framework, allowing them to build flexible pipelines with minimal effort.

Confluent 3.1, the latest release of the stream processing platform built on Apache Kafka, introduces several enhancements aimed at improving enterprise readiness, including extended features in the Confluent Control Center, Automatic Data Balancing to ensure efficient workloads across clusters, and Multi-datacenter replication for easier management of multiple Kafka clusters. This version also includes new connectors for Elasticsearch and JDBC Sink, a high-performance Go client, and the latest Kafka release 0.10.1.0, addressing over 200 bug fixes and improvements. The update enhances security with enterprise authentication integration and stream monitoring capabilities, while continuing to support users with expert training and technical consulting. Users are encouraged to explore the new features and provide feedback through the public Confluent Platform mailing list.

The Apache Kafka community has released version 0.10.1.0, which includes several major improvements and new features. Jiangjie (Becket) Qin has been invited to join the PMC as a committer, bringing new perspectives to the project. Various improvement proposals are being discussed and voted on, addressing issues such as memory usage, authentication mechanisms, and configuration flexibility. These changes aim to enhance the performance, security, and usability of Kafka, while also providing developers with more options for deployment and management. The community is actively exploring ways to improve the platform, including limiting incoming requests, bounding fetch memory usage, supporting SASL/SCRAM mechanisms, and adding a compaction tombstone flag.

Apache Kafka, originally developed at LinkedIn with a focus on Java, has evolved to support multiple programming languages through a versioned, binary protocol, allowing independent client development. Initially, the quality of these non-Java clients varied, impacting user experiences outside the Java ecosystem. Confluent, recognizing the need for high-quality clients across languages, introduced an open-source REST layer for Kafka and simplified the Kafka consumer protocol, integrating it into the core Kafka architecture to ease client development and scalability. The strategy involved leveraging the high-quality C client, librdkafka, for developing native clients in various languages, such as Python and Go, ensuring feature parity and robust performance. This approach, favoring wrapping existing C code over creating new clients from scratch, enhances client reliability and efficiency across platforms. The recent release of Confluent Platform 7.7 introduces key features such as enhanced security with OAuth support and integration with Apache Flink® and Amazon OpenSearch, emphasizing Confluent's commitment to making Kafka a versatile platform for stream processing across diverse programming environments.