January 2019 Summaries

The text highlights the increasing importance of event-driven architecture (EDA) in modern business environments, emphasizing the need for real-time data processing to address the challenges posed by global, cloud-native applications. Traditional architectures are deemed insufficient for the demands of real-time, scalable operations, prompting a shift towards EDA, which enables companies to ingest, process, store, and react to data as it is generated. This approach is essential for evolving business needs, offering benefits like elasticity, decoupling, and the ability to adapt to changes without disrupting existing systems. The text also underscores the paradigm shift from command-based interactions to event-first thinking, where events are treated as immutable facts that drive reactions and system evolution. It argues that adopting an event-driven model, despite its initial complexity, provides a flexible and robust framework for building scalable, resilient applications that can seamlessly integrate with modern technologies like IoT and machine learning. The document concludes by promoting further exploration of event-driven architectures and outlines resources for learning and implementing such systems effectively.

Kafka Summit NYC 2019 is set to return in April, offering a platform for discussing and learning about Apache Kafka and streaming technologies. The event will feature keynotes from notable figures such as Pivotal's James Watters and Confluent's Jay Kreps, with a revised track structure to accommodate the evolving streaming landscape. Attendees can explore sessions that delve into Kafka's core aspects, real-time computation techniques, and event-driven development. The summit will also present real-world use cases demonstrating the application of streaming platforms across various industries. The agenda promises a robust lineup of talks, underscored by a competitive selection process, thanks to a dedicated Program Committee. Additionally, the event will offer networking opportunities and insights from a diverse range of industry experts, encouraging early registration with special discounts.

Confluent Cloud is 100% compatible with Apache Kafka, allowing users to utilize the same admin tools that come with the community edition of Apache Kafka. To access these tools in Confluent Cloud, users need to generate a configuration properties file by following specific steps outlined in this blog post. This file includes connectivity details such as bootstrap server endpoints and key/secret pairs for authentication. The file is used by various admin tools like kafka-console-producer and kafka-console-consumer to establish connectivity with the cluster. Additionally, users can troubleshoot potential issues related to records consumption within consumer groups using the kafka-consumer-groups tool. Connectivity from user's end to Confluent Cloud clusters is necessary, which may involve verifying network topology and firewall rules. Users have free-of-charge access to components of Confluent Platform, allowing them to leverage these tools with Kafka along with other services like Confluent Control Center and ksqlDB.

Confluent, a company focused on event streaming platforms, has raised $125 million in Series D funding led by Sequoia with participation from Index Ventures and Benchmark. The company's cloud offering went GA on both AWS and Google Cloud Platform, and it grew its subscription bookings 3.5x year-over-year. Confluent believes that event streaming platforms will play a foundational role in the architecture and operation of modern digital companies. It is currently working to expand its technology and partner with more customers adopting this approach. The company sees an increasing demand for data streaming platforms as businesses realize their importance in today's world.

Apache Kafka provides a rich set of APIs for developing real-time applications, Confluent Cloud offers infrastructure management offloading, and validation is crucial to ensure the application works as designed. Developers can unit test their Kafka Streams applications using kafka-streams-test-utils, TopologyTestDriver, MockProcessorContext, MockProducer, and MockConsumer utilities. Integration testing validates that individual components work together seamlessly, while simulated environments like EmbeddedKafkaCluster allow for parallelism and fault injection capabilities. Confluent Schema Registry helps manage schemas and provides compatibility checks to ensure the contract between producers and consumers is not broken. Testing applications in realistic deployment scenarios with Confluent Cloud instances or locally using Docker setups can provide additional test coverage. The ultimate goal is to help businesses succeed with Kafka, and this blog post guides developers through testing streaming applications at all levels from unit testing to multi-datacenter environments.

This webinar will cover how to build RAG-enabled GenAI with Confluent, Flink & MongoDB by leveraging stream processing technology, which is used to collect, store and manage continuous streams of data. Stream processing has numerous use cases and provides benefits such as decoupling dependencies between services, providing pluggability, enabling services to evolve independently, etc. The webinar will introduce a new resource, a free self-paced tutorial for developers who are just getting started with stream processing, which covers the basics of the Kafka Streams API and common patterns to design and build event-driven applications. The tutorial is based on a small microservices ecosystem showcasing an order management workflow, where business events propagate through the ecosystem, triggering services to validate orders in parallel. The system also includes a blocking HTTP GET interface for clients to read their own writes, and other services such as sending emails and collating orders. The tutorial provides exercises for developers to learn patterns for writing solid streaming applications and gain experience with using the Kafka Streams API.

To test against a real Apache Kafka cluster, developers can use the Confluent CLI to launch Confluent Platform and produce/consume data from Kafka topics. However, generating realistic test data for these topics can be challenging. To overcome this, Confluent provides a Kafka Connect Datagen Connector that allows developers to generate more interesting test data locally. This connector can produce records with complex data types, randomizing the data and customizing the schema as needed. Additionally, developers can use predefined datasets or define their own schema specifications to generate specific data formats such as Avro or JSON. With this tool, developers can exercise their client applications, build demos, troubleshoot issues, or learn more about how Kafka works in a controlled environment.