June 2023 Summaries

The second annual Data Streaming Awards invited public nominations for organizations and individuals using data streaming technology to improve business operations and customer experiences, with winners announced at Current 2023. Awards were offered across six categories, including innovation, company and startup of the year, individual systems, industry legends, and company-wide implementations, and nominations were accepted anonymously through July 17, 2023. An independent and diverse judging panel evaluated entries for innovation, impact, and broad interest, while organizers emphasized transparency through open submissions, confidential voting, and limits on Confluent-employed judges. The announcement also highlighted winners from the inaugural 2022 awards, including NASA/JPL, Walmart, and LinkedIn.

Confluent has been recognized as Microsoft's OSS on Azure Partner of the Year for the United States, highlighting its excellence in delivering open source-based applications and infrastructure solutions on Microsoft Azure. This award underscores the benefits of integrating Azure services with open-source software like Confluent, which simplifies the deployment and integration of Apache Kafka® on Azure. Confluent's cloud-native, fully managed solution, powered by the Kora engine, is noted for being more scalable, reliable, and performant than Apache Kafka, and it offers private connectivity through Azure Private Link. With over 120 pre-built connectors, users can easily build real-time streaming pipelines to various Azure services. Confluent's solution is accessible through Azure Marketplace, offering features like enterprise-grade security, management, and a seamless onboarding process with a 30-day trial. This recognition, marking Confluent's third Partner of the Year award, reflects its commitment to advancing data streaming as a fundamental aspect of modern business on Azure.

The concept of events has been explored in various contexts within the Apache Kafka ecosystem, including event-driven design, event sourcing, and stream processing. Events are representations of facts or things that happen, and their role can vary depending on the context. Designing events involves carefully choosing a conceptual model for event representation based on the role the event plays, while event streaming involves aggregating, filtering, and joining multiple streams in real-time. Event-driven design refers to building architecture that's aware of events using patterns such as event notification, event-carried state transfer, and command query responsibility segregation. These concepts are crucial for designing reactive data structures and building applications that can handle real-time data pipelines and applications. Understanding these ideas is essential for developers looking to build scalable and efficient systems.

Confluent Cloud` is being used by `Mobile Premier League` (`MPL`) as their central nervous system for data streaming and sharing, enabling the processing of hundreds of millions of events per day. With this platform, MPL can deliver exceptional player experiences, reduce player churn, and increase revenue generation. The solution uses a fully managed streaming platform to stream real-time events from various sources, such as gaming apps, relational databases, and marketing SaaS apps, into target systems for analysis and processing. This allows MPL to optimize matchmaking, onboarding personalization for new players, and real-time fraud detection and prevention, ultimately creating a competitive edge in the eSports and mobile gaming industry.

Extend, a product protection platform serving merchants in industries such as automotive, electronics, jewelry, and furniture, adopted Confluent Cloud and AWS to support real-time product catalog updates and transition from SNS/SQS-based asynchronous messaging to a more scalable event-driven architecture. Its earlier serverless-first AWS approach reduced infrastructure management during its MVP phase, but increasing transaction volumes and seasonal spikes exposed limitations in throughput, event ordering, and duplication handling. Confluent’s fully managed Kafka service provided ordered, replayable, retained events while removing the need for engineers to manage clusters, brokers, networks, and other Kafka infrastructure. Extend automated resource provisioning, permissions, access controls, and topic creation through CloudFormation, CDK, Confluent APIs, internal libraries, and AWS Lambda triggers, allowing engineers to build new data streams quickly without long-lived infrastructure. The implementation improved scalability, operational efficiency, security through least-privilege access, and time to market, while reducing total cost of ownership and supporting future data mesh initiatives and integrations with merchant platforms such as NetSuite, Shopify, and BigCommerce.

Wealthsimple uses Confluent’s managed Apache Kafka service to centralize transaction data from its microservices into a real-time general ledger, supporting investment, trading, cryptocurrency, and banking products. Its Book of Record team ingests transaction events from multiple services through Kafka topics, enabling downstream systems to reconcile balances and synchronize customer-facing and back-end processes. The company selected Confluent over self-managed Kafka to avoid cluster-management overhead and allow engineering teams to focus on financial product development, while Schema Registry provides evolving API contracts and standardized data quality. Replacing polling with event streaming increased one key service’s capacity from 2,000 to more than 18,000 order fills per minute, and Confluent has also provided scalable topics, replayable data, and managed support. Wealthsimple plans to migrate further Amazon SQS workloads, develop real-time fraud analytics, and expand organization-wide use of streaming data for the general ledger, while emphasizing training and data-governance practices for effective schema management.

The release of Apache Kafka® 3.5.0 introduces numerous enhancements and updates, including early access to a feature that allows Kafka clusters to migrate from Zookeeper to KRaft mode without downtime, though it is currently recommended for non-production environments. Notably, ZooKeeper is deprecated in this version and will be removed in Apache Kafka 4.0. The release includes several Kafka Improvement Proposals (KIPs), such as rack-aware partition assignment for consumers, a new ConfigProvider to use environment variables, enhancements to support SCRAM for inter-broker authentication, and improvements in handling serialization exceptions. Additionally, versioned state stores are introduced for better accuracy in processing out-of-order records, a Boolean Serde is added to the public interface, and first-class offsets support is introduced in Kafka Connect. The release also improves the dedicated mode of MirrorMaker 2.0 by supporting multiple instances and automatic reconfigurations, with future updates expected to expand REST API capabilities for managing offsets. This release was a collaborative effort involving contributions from numerous authors, reflecting the community's engagement and support.

Caroline Staudenraus transitioned from elementary literacy teaching through AmeriCorps to technology sales at Confluent after proactively seeking a referral for an SDR role in 2018. Over five years, she advanced through five positions to become an account executive in the digital native commercial segment, where she develops new business opportunities while supporting existing customers alongside cross-functional teams. Her hybrid Austin-based role varies between customer support, prospecting, product learning, and closing deals, and she is currently adapting her sales approach to an economic downturn marked by longer purchasing approval processes. Caroline credits Confluent’s collaborative “shoulder-tapping” culture, in which colleagues readily share knowledge and support one another’s growth, for helping her build technical expertise and progress professionally, while noting that the company’s rapid growth can make it difficult to communicate changing policies and processes consistently.

The text discusses how to build GenAI apps faster using ChatGPT and event streaming. The fundamental obstacle is that ChatGPT can't provide personalized answers to customer queries without access to the company's internal data stores. To overcome this, companies need to safely provide timely data from their internal data stores to ChatGPT. Event streaming is a good solution to bring all of these systems together, allowing companies to construct a unified view of each customer that's easy to query with low latency. The architecture involves using event streams to process raw information and then sink it into a relational database or document store. A retrieval-augmented generation (RAG) approach is used to connect policy data to ChatGPT by making an embedding out of the prompt itself and querying a vector database for related information. Plugins are also used to extend ChatGPT's capabilities, such as the Retrieval Plugin which acts as a proxy layer between ChatGPT and the vector database. The approach has limitations, including reliance on large context windows and vulnerability to prompt injection attacks. However, it enables companies to build real-time support agents that can take real action on their behalf.

Confluent's Terraform provider has become integral to continuous integration pipelines in enterprises with over 3 million downloads since its release in June 2022. HashiCorp's Terraform enables changes to be made to Confluent infrastructure via declarative merge requests, a GitOps approach that provides enhanced security, improved collaboration, and increased operational efficiency. Policy-as-code is used to protect against deploying changes that violate organization's security and auditing requirements, with validation performed for every proposed change to ensure compliance with established policies. The Terraform plan phase generates a plan file in JSON format, which is then validated against established policies using HashiCorp's Sentinel. If the proposed plan meets all of the policies, Terraform can move on to the apply phase and automatically create the cluster, while policy violations trigger potential actions such as advisory, soft-mandatory, or hard-mandatory responses. By utilizing GitOps and policy-as-code, users can propose changes with confidence that they won't accidentally violate important security requirements.