May 2017 Summaries

In the context of evolving service architectures, the text explores the shift from traditional request-driven microservices, which rely heavily on synchronous interactions, to event-driven approaches that reduce service coupling by leveraging asynchronous event streams. This paradigm shift allows for greater flexibility, pluggability, and independence among services, enabling them to progress business goals through cascades of events rather than direct commands. The text emphasizes the advantages of event-driven systems, such as reduced coupling, the ability to query local views constructed from event streams, and the potential for services to evolve independently while maintaining consistency through assigned responsibilities for event propagation. By integrating concepts like Receiver Driven Routing and Event-Carried State Transfer, the architecture accommodates complex ecosystems with a backbone of events, which can be further optimized using modern streaming platforms like Kafka. These patterns are presented as guiding principles rather than rigid rules, allowing for adaptability in architectural design and paving the way for scalable, highly-available service infrastructures.

The text announces the launch of Confluent Cloud, a scalable streaming platform based on Apache Kafka, aimed at cloud-first developers and enterprises transitioning to the cloud, with early access available before its general release later in the year. The announcement was made at the Kafka Summit in NYC, with plans for the next summit in San Francisco featuring a robust schedule of speakers. The text also discusses the upcoming release of Kafka 0.11.0.0 and highlights the recent release of Kafka 0.10.2.1, which includes important bug fixes. It further mentions the 2017 Apache Kafka report, illustrating the widespread adoption of Kafka in large enterprises for various uses such as stream processing and data integration, and concludes with a note on the release of Apache Kafka 3.8.0, which includes numerous new features and improvements. Additionally, a Kafka-themed coding project about the tabs versus spaces debate is briefly mentioned.

The Kafka Streams API allows developers to build real-time data processing applications with Apache Kafka without requiring a separate processing cluster. This enables fast development and deployment of applications, regardless of the programming language used. The Confluent demo applications and Docker-based demo setup are available for users to get started quickly. Container technologies like Docker can be used to speed up iterative development on laptops, and the Kafka Streams API can be containerized and deployed alongside other services using Confluent Docker images. This allows developers to use the same organizational processes and technical tooling for development, testing, packaging, deployment, and monitoring of Kafka Streams applications as they do elsewhere in their company.

Kafka Summit New York City was a significant gathering of the Apache Kafka community, featuring over 500 participants from various industries. The event focused on the emergence of event streaming platforms, with Kafka at its core. Confluent announced Confluent Cloud, an Apache Kafka as a Service offering that aims to make it easy for anyone to implement and scale the leading event streaming platform without cloud provider lock-in. Other highlights included keynotes from ING's Ferd Scheepers on Kafka's role in modernizing data strategy and customer experience, and discussions on exactly once semantics and upcoming Kafka Improvement Proposals. The next Kafka Summit will take place in San Francisco on August 28th.

Apache Kafka is a powerful enterprise streaming platform that offers seamless integration for client applications and simplifies operational tasks such as load distribution, data transfer, and partition management. While Kafka's default settings allow for straightforward development and testing, tuning its configuration parameters is essential to meet specific service goals like throughput, latency, durability, and availability, which often involve tradeoffs. A comprehensive white paper provides guidance on optimizing Kafka deployments by detailing configuration parameters for producers, brokers, and consumers, and emphasizes the importance of benchmarking to tailor settings for particular use cases and hardware profiles. The paper also highlights the significance of monitoring performance with tools like the Confluent Control Center and the value of professional services from Confluent. Additionally, the release of Apache Kafka 3.8.0 introduces new features and improvements, with further details available in the release notes.

The Kafka Summit NYC, held at the Midtown Hilton, celebrates the innovative use of Apache Kafka and streaming data by leading companies such as ING, Uber, and The New York Times to transform business operations. With over 500 attendees and 19 sponsors, the summit highlights keynotes from notable figures like Jay Kreps and ING's Chief Information Architect, Ferd Scheepers, emphasizing Kafka's role in ING's digital transformation. The event also features a hackathon focused on microservices implemented with Kafka Streams API, while those unable to attend can access live streams and session videos online. The summit underscores the importance of cloud services and data in enhancing government agency operations, as demonstrated by collaborations like Confluent and AWS.

Confluent Cloud, a fully managed streaming data service, offers a simplified and efficient way to operate Apache Kafka in the public cloud, aiming to serve as a central data backbone for companies. By eliminating the complexities of managing Kafka’s infrastructure, it allows developers to focus on application development instead. As the original creators of Apache Kafka, Confluent leverages extensive operational expertise to provide a service that requires minimal setup and maintenance, offering features such as automatic scaling, security, and compatibility with open-source Kafka tools. Confluent Cloud supports major cloud providers like AWS, and plans to include Azure and Google Cloud, ensuring flexibility and freedom from vendor lock-in. This launch signifies a significant advancement for cloud-first developers and enterprises shifting to the cloud, furthering Confluent's mission to integrate streaming platforms at the core of business operations. An early access program is available for users eager to adopt this service, and an upcoming online talk will provide additional insights into building portable and robust streaming applications with Confluent Cloud.

Kafka Summit NYC is an upcoming event featuring companies like Google, Uber, and AltspaceVR sharing their experiences with stream processing solutions. The Streams Track will explore new Kafka features for building event-driven applications and microservices, as well as how companies use Kafka in their stream processing architectures. Some highlighted sessions include "Hanging Out With Your Past Self In VR: Time-Shifted Avatar Replication Using Kafka Streams" by AltspaceVR's Greg Fodor, "The Data Dichotomy: Rethinking Data & Services with Streams" by Confluent's Ben Stopford, and "Scalable Real-time Complex Event Processing at Uber" by Shuyi Chen. The event will also include a Kafka Summit Hackathon to help the community learn how to build streaming applications with Apache Kafka.

Confluent conducted a survey of over 350 organizations across 47 countries, finding that 86% are increasing their use of Apache Kafka, with many using it in the public cloud and for stream processing and data integration. The survey also highlighted growing adoption of microservices, IoT, and monitoring, with companies using Kafka to build data pipelines and core business applications. Confluent has introduced a Kafka certification program to address talent shortages, and a new release of Apache Kafka 3.8.0 is available, featuring several new features and improvements. With the growth of streaming platforms like Kafka, there is an increasing need for experts in these technologies, as companies are adopting microservices and building distributed architectures.

The Google Dataflow team has developed a model for handling time in stream processing that recognizes the challenge of globally ordering data arrival, leading to the integration of a system for coping with this issue into the Beam API. This approach is widely adopted by other stream processing systems. A key observation is that most computations can be handled using tables, which represent immutable entities, but many organizations have mutable databases at their core. To address this, a table representation for data stores is introduced, allowing streaming apps to combine pure events with data from these sources. This enables the generalization of the Dataflow model for handling windowed computation, where computations are represented as tables with aggregation IDs and window times. The key benefit of this approach is that it allows for semantic processing of output streams, enabling the modeling of windowed computation without relying on complex watermark systems. Additionally, the frequency at which output is emitted can be tuned to balance update lag and output volume, offering a practical trade-off between these non-functional parameters.