September 2019 Summaries

Confluent Cloud offers a serverless experience for Apache Kafka on various cloud platforms, eliminating the need for infrastructure configuration and management, and providing true elastic scaling without manual cluster sizing adjustments. With consumption-based pricing, users only pay for what they use, and with an uptime guarantee of 99.95%, downtime due to administrative issues is minimized. Support options range from free community support to paid plans, allowing organizations to choose the level of support that suits their needs, and Confluent Cloud Enterprise provides a custom setup for mission-critical applications. By leveraging Confluent Cloud, organizations can simplify their Kafka environment and focus on building event-driven applications.

The text provides a lively recount of a Kafka Summit event, highlighting its growth from a modest gathering in 2017 to a large, vibrant community event in San Francisco. The summit featured notable speakers like Jun Rao, co-creator of Apache Kafka and co-founder of Confluent, who shared insights into Kafka's origins and its significant adoption among Fortune 500 companies. The event included engaging sessions, such as a Q&A with Walmart's VP of Cloud, Chris Kasten, discussing Walmart's digital transformation strategies. Attendees enjoyed various activities, including a Build-A-Bear session with Confluent Community Catalysts and on-demand donuts. The text also promotes upcoming events and resources, such as a keynote by Confluent CEO Jay Kreps and a livestream covering AI innovations, encouraging readers to explore further learning opportunities at future conferences like AWS re:Invent 2024.

Confluent Platform 5.4 introduces robust data governance support through Schema Validation, enabling enforcement of data correctness at the broker level by ensuring that data produced to Apache Kafka topics adheres to a registered schema ID in the Confluent Schema Registry. This approach marks a shift from treating schemas as mere agreements between clients to a more comprehensive governance mechanism integrated into the event streaming platform itself, facilitating centralized policy enforcement. By configuring the `confluent.schema.registry.url` in the server properties, users can enable Schema Validation at the topic level, ensuring that both key and value schemas are validated upon data entry. This feature not only prevents data corruption but also represents a foundational step towards advanced data governance capabilities, aligning with the broader enterprise need for structured data management across complex, real-time systems. Additionally, the update highlights the compatibility and migration possibilities between Confluent Server and Kafka, with an emphasis on the potential overhead of message-by-message schema validation and the flexibility provided by configuring schema validation strategies.

The text discusses building real-time interactive queries on large datasets using Apache Kafka and Rockset, a scalable SQL search and analytics engine. It highlights the benefits of using Kafka as an event streaming platform for mission-critical infrastructures and core operations platforms, leveraging its ability to handle massive volumes of data in real-time. The article showcases how Rockset can be used to perform interactive analytics on top of Apache Kafka, providing low-latency queries and fast SQL queries on raw data coming from Kafka through an API. It also demonstrates how to integrate Kafka with other valuable components like Confluent Schema Registry and ksqlDB for continuous stream processing on data streams using simple SQL commands. The text emphasizes the importance of leveraging cloud elasticity for distributed query processing, ensuring reliable performance at scale without managing shards or servers. Finally, it showcases a step-by-step example of creating a real-time monitoring dashboard on a Twitter JSON feed in Kafka, joined with more detailed company information from Nasdaq, and connected to Redash for interactive visualization.

The concept of software is eating the world, transforming how businesses operate and interact with customers. Traditional database architectures are being reevaluated as companies shift towards building whole parts of their business in code. The rise of event streams presents a new paradigm for thinking about data, where data is treated as a continuous, active stream of events, fed to passive queries that react and process the stream. Confluent's mission is to build an event streaming platform that incorporates storage and query processing capabilities, enabling companies to harness real-time data and drive business decisions. The event streaming platform has significant implications for how companies are structured, with the potential to become a central nervous system for software-defined companies.

[Incremental Cooperative Rebalancing] in Apache Kafka clients improves scalability, reduces rebalancing costs, and increases overall throughput by allowing workers to run tasks without disruptions, resulting in a more efficient and flexible load balancing mechanism. This new approach enables large-scale deployment of clients, alleviates the impact of existing stop-the-world rebalancing protocols, and provides a significant improvement in terms of cost, scaling, and performance. The Incremental Cooperative Rebalancing protocol is implemented in Kafka Connect, allowing connectors to scale beyond current limits and enabling more centralized and manageable connector deployments.

Kafka Summit San Francisco is a highly anticipated event that brings together experts in the field of Apache Kafka®. To make the most out of this conference, attendees are advised to familiarize themselves with the schedule and attend a diverse range of sessions. It's also important to network with fellow attendees and engage with speakers during breaks or informal settings. Attendees should arrive early, stay hydrated, and take breaks as needed to fully absorb the wealth of information presented at the conference. The event takes place at Hilton San Francisco Union Square from September 30th to October 1st, with a half-day tutorial on September 29th. Don't miss the last week to register using the code blog19 for a 30% discount.

The text provides an in-depth examination of managed services for Apache Kafka, particularly focusing on the distinction between fully managed and partially managed solutions. It highlights the complexities of deploying and managing Kafka, whether on-premises or in the cloud, and the benefits of using managed services like Confluent Cloud, which offers a serverless experience and relieves users from the operational burdens of running Kafka clusters. The importance of elasticity, interoperability, and avoiding vendor lock-in is emphasized, as these factors influence the ease of scaling and migrating applications across different cloud providers. Confluent Cloud is presented as a comprehensive solution that supports major cloud providers and offers additional tools like Kafka Connect, ksqlDB, and Schema Registry to simplify event streaming application development. The text also touches on the challenges of maintaining expertise in the rapidly evolving Kafka ecosystem and how managed services backed by experts can provide stability and support.

The text discusses the evolution of real-time data pipelines and their significance in today's business landscape. It highlights how traditional ETL processes have evolved from overnight batches to more frequent ones, but eventually reach a point where they show signs of strain. Real-time data pipelines, enabled by event streaming platforms like Apache Kafka®, allow businesses to deliver insights based on current events within seconds, as opposed to minutes or hours. The text provides examples of real-world applications such as fraud detection in finance and retail industries, and the transformation of Express Scripts' data architecture from a mainframe to a low-latency pipeline using Confluent Platform. It also promotes Kafka Summit as an event where developers share their experiences with real-time pipelines and how they fit into broader business concerns.

Confluent Platform 5.4 introduces Multi-Region Clusters, enhancing Apache Kafka's capabilities for multi-datacenter configurations by incorporating features such as Follower Fetching, Observers, and Replica Placement. This update allows for synchronous or asynchronous data replication on a per-region basis, improving data durability and simplifying disaster recovery operations. Follower Fetching reduces cross-WAN traffic by enabling consumers to read from local replicas, while Observers act as asynchronous replicas that enhance data locality and disaster recovery without impacting durability. The new JSON-based replica placement strategy further refines replica assignments, enabling precise control over data distribution across regions. Additionally, the release addresses potential latency issues with ZooKeeper in multi-regional setups and highlights future improvements with the anticipated removal of ZooKeeper, as discussed in the KIP-500 proposal. These advancements aim to streamline infrastructure operations and reduce the operational overhead associated with running Kafka across multiple data centers.

The text discusses how technology and infrastructure changes can impact software architecture beyond just technical aspects. It highlights the growing adoption of Apache Kafka as an event streaming platform, enabling companies to build scalable systems with microservices exchanging inputs and outputs in real-time through Kafka topics. The author shares examples from Ticketmaster's transition to an event streaming architecture using Kafka and how it has helped them manage complexities in their system. Additionally, the text mentions that Kafka not only helps build scalable systems but also transforms the way architects think about those systems, moving away from large monolithic programs to evolvable services maintaining their own state and communicating through scalable logs.

The text delves into the concept of event sourcing, particularly derivative event sourcing, and its application in complex systems with multiple order services, such as those at the Cabot Cove Detective Agency. It explores the challenges of integrating legacy systems with modern data streaming solutions like Apache Kafka, especially when dealing with technical debt and diverse technical environments. The text explains how change data capture (CDC) can be used to derive events from existing databases, allowing for seamless integration across various services. It highlights the use of tools like Oracle GoldenGate and Debezium to facilitate this process and describes how Kafka Streams can transform and aggregate these events into a central event service. The piece emphasizes the importance of designing flexible event schemas and the benefits of decoupling event processing, which can enhance system resilience and scalability. It concludes by inviting readers to explore more complex scenarios and best practices in derivative event sourcing through additional resources and talks.

The text provides a detailed guide on how to integrate Confluent Schema Registry and Avro serialization with Spring Boot applications using Apache Kafka. It outlines the process of generating a project starter with necessary dependencies such as Kafka Schema Registry Client and Avro, and provides code snippets for creating producers, consumers, and REST controllers to handle data in Avro format. The guide includes instructions on setting up a local Confluent Platform environment, compiling the application, and running it both locally and on Confluent Cloud, offering configuration details for cloud deployment. Additionally, it touches upon the recent release of Confluent Platform 7.7, highlighting features like enhanced security and integration capabilities with Apache Flink and Amazon OpenSearch. The text concludes by encouraging readers to explore more Kafka Tutorials for further learning and offers a promo code for Confluent Cloud services.