June 2023 Summaries

Confluent has been recognized as Microsoft's OSS on Azure Partner of the Year for the United States, highlighting its excellence in delivering open source-based applications and infrastructure solutions on Microsoft Azure. This award underscores the benefits of integrating Azure services with open-source software like Confluent, which simplifies the deployment and integration of Apache Kafka® on Azure. Confluent's cloud-native, fully managed solution, powered by the Kora engine, is noted for being more scalable, reliable, and performant than Apache Kafka, and it offers private connectivity through Azure Private Link. With over 120 pre-built connectors, users can easily build real-time streaming pipelines to various Azure services. Confluent's solution is accessible through Azure Marketplace, offering features like enterprise-grade security, management, and a seamless onboarding process with a 30-day trial. This recognition, marking Confluent's third Partner of the Year award, reflects its commitment to advancing data streaming as a fundamental aspect of modern business on Azure.

The concept of events has been explored in various contexts within the Apache Kafka ecosystem, including event-driven design, event sourcing, and stream processing. Events are representations of facts or things that happen, and their role can vary depending on the context. Designing events involves carefully choosing a conceptual model for event representation based on the role the event plays, while event streaming involves aggregating, filtering, and joining multiple streams in real-time. Event-driven design refers to building architecture that's aware of events using patterns such as event notification, event-carried state transfer, and command query responsibility segregation. These concepts are crucial for designing reactive data structures and building applications that can handle real-time data pipelines and applications. Understanding these ideas is essential for developers looking to build scalable and efficient systems.

Confluent Cloud` is being used by `Mobile Premier League` (`MPL`) as their central nervous system for data streaming and sharing, enabling the processing of hundreds of millions of events per day. With this platform, MPL can deliver exceptional player experiences, reduce player churn, and increase revenue generation. The solution uses a fully managed streaming platform to stream real-time events from various sources, such as gaming apps, relational databases, and marketing SaaS apps, into target systems for analysis and processing. This allows MPL to optimize matchmaking, onboarding personalization for new players, and real-time fraud detection and prevention, ultimately creating a competitive edge in the eSports and mobile gaming industry.

The release of Apache Kafka® 3.5.0 introduces numerous enhancements and updates, including early access to a feature that allows Kafka clusters to migrate from Zookeeper to KRaft mode without downtime, though it is currently recommended for non-production environments. Notably, ZooKeeper is deprecated in this version and will be removed in Apache Kafka 4.0. The release includes several Kafka Improvement Proposals (KIPs), such as rack-aware partition assignment for consumers, a new ConfigProvider to use environment variables, enhancements to support SCRAM for inter-broker authentication, and improvements in handling serialization exceptions. Additionally, versioned state stores are introduced for better accuracy in processing out-of-order records, a Boolean Serde is added to the public interface, and first-class offsets support is introduced in Kafka Connect. The release also improves the dedicated mode of MirrorMaker 2.0 by supporting multiple instances and automatic reconfigurations, with future updates expected to expand REST API capabilities for managing offsets. This release was a collaborative effort involving contributions from numerous authors, reflecting the community's engagement and support.

The text discusses how to build GenAI apps faster using ChatGPT and event streaming. The fundamental obstacle is that ChatGPT can't provide personalized answers to customer queries without access to the company's internal data stores. To overcome this, companies need to safely provide timely data from their internal data stores to ChatGPT. Event streaming is a good solution to bring all of these systems together, allowing companies to construct a unified view of each customer that's easy to query with low latency. The architecture involves using event streams to process raw information and then sink it into a relational database or document store. A retrieval-augmented generation (RAG) approach is used to connect policy data to ChatGPT by making an embedding out of the prompt itself and querying a vector database for related information. Plugins are also used to extend ChatGPT's capabilities, such as the Retrieval Plugin which acts as a proxy layer between ChatGPT and the vector database. The approach has limitations, including reliance on large context windows and vulnerability to prompt injection attacks. However, it enables companies to build real-time support agents that can take real action on their behalf.

Confluent's Terraform provider has become integral to continuous integration pipelines in enterprises with over 3 million downloads since its release in June 2022. HashiCorp's Terraform enables changes to be made to Confluent infrastructure via declarative merge requests, a GitOps approach that provides enhanced security, improved collaboration, and increased operational efficiency. Policy-as-code is used to protect against deploying changes that violate organization's security and auditing requirements, with validation performed for every proposed change to ensure compliance with established policies. The Terraform plan phase generates a plan file in JSON format, which is then validated against established policies using HashiCorp's Sentinel. If the proposed plan meets all of the policies, Terraform can move on to the apply phase and automatically create the cluster, while policy violations trigger potential actions such as advisory, soft-mandatory, or hard-mandatory responses. By utilizing GitOps and policy-as-code, users can propose changes with confidence that they won't accidentally violate important security requirements.