November 2023 Summaries

This blog post discusses how businesses can leverage real-time data streaming and processing technologies to drive value using AI/ML models. It highlights the importance of streaming analytics infused with AI and ML in various domains, such as fraud prevention, autonomous vehicles, voice assistants, recommendation engines, dynamic pricing, predictive maintenance, and intelligent cybersecurity systems. The post also emphasizes the role of data streaming platforms in powering modern AI/ML models with real-time, trusted data across the enterprise. It further explains how Confluent Cloud simplifies the integration of streaming data into AI/ML models by abstracting away technical complexity and supporting various features like model deployment, monitoring, and embedding models into a real-time stream. The post concludes that real-time AI/ML is becoming a strategic differentiator as it increasingly becomes important to incorporate real-time context from streaming data into the decision-making process.

librdkafka, the widely used Kafka client library for C, C++, Python, .NET, Go, and other non-JVM environments, uses a production model that differs substantially from the Java client and is especially relevant to external-facing applications that must tolerate Kafka outages. Applications configure a producer and delivery-report callback, submit messages with produce APIs, and explicitly call rd_kafka_poll() to process broker responses and release message memory. Sending involves application threads that enqueue messages, a main library thread that retrieves cluster metadata, assigns partitions, and creates broker threads, and one broker thread per broker that batches, serializes, transmits, retries eligible failures, and returns results through a reply queue. Delivery callbacks provide error and persistence-status information, including whether a message was not persisted, possibly persisted, or confirmed persisted, with the latter most reliable when acks=all is used. The library’s built-in retries and idempotent production are generally preferable to application-level retries because they better preserve ordering and avoid duplicate records. Since messages remain in memory until replies are polled and callbacks execute, queue size limits can cause produce calls to return a queue-full error, requiring applications to poll and then retry submission.

The financial industry is evolving to adopt modern core banking systems that can respond and scale to meet customer demand in near-real time. Traditional mainframe-based architectures are being replaced by event-driven streaming data platforms, which enable banks to unlock their critical mainframe data, minimize disruptions, and power their ever-growing landscape of cloud-native applications and systems. The key technologies used in the modern core banking stack include unlocking core banking system of record, enriching data in real-time with Apache Flink, decoupling system of record and system of access with MongoDB, and writing data out to a non-relational DB like Mongo Atlas. Mainframe migrations are being done incrementally, often through connectors like MQ on Z and Oracle CDC, which reduce costs by 90% or more, depending on the solution. The modernization process is slow, iterative, and typically not big bang, with many large financial institutions still in the early innings of this modernization.

Ness Digital Technologies partnered with a large U.S. equity-derivatives clearing organization to modernize its aging, batch-based risk and valuation systems as growing trading volumes, market volatility, and regulatory demands made near-real-time processing increasingly necessary. Under the multi-year Renaissance Initiative, the organization replaced core clearing, data, and risk components with a modular, event-driven architecture using Confluent Cloud and Apache Kafka alongside AWS, Snowflake, S3, Redshift, and Athena. The streaming platform connects data centers and cloud systems, supports encrypted and potentially anonymized data flows, and enables horizontal scaling rather than reliance on larger centralized database infrastructure. According to the account, the new system processes roughly 10,000 messages per second, supports price-change calculations and re-aggregations at 5-, 20-, and 60-minute intervals, improves margining, stress testing, back testing, auditability, and clearing-member visibility, and is intended to help the organization introduce new products and models more quickly while meeting stricter risk-management and regulatory requirements.

Navigating the complexities of Apache Kafka can be challenging, but a variety of resources from Confluent can facilitate a smoother learning experience. With a rich library of educational content, beginners can access tutorials on building Kafka producer and consumer applications, as well as understanding core concepts such as partitioning, replication, and Kafka Streams. Notable resources include Tim Berglund's YouTube series, which breaks down Kafka fundamentals and provides hands-on exercises, and the Kafka Streams 101 course, which covers advanced topics like handling out-of-order data. Tutorials offer practical, step-by-step guidance for building applications in cloud-based or standalone environments, accommodating various programming languages such as Java, JavaScript, Python, and .NET/C#. Additionally, community support is available through Slack and forums, helping learners overcome challenges and further explore Kafka's versatility in event-driven design and data streaming.

The post explains how to stream Kafka data from Confluent Cloud to end-user applications in real time through Ably’s Kafka Connector, combining Confluent Cloud’s managed Kafka processing, governance, and fault tolerance with Ably’s global edge delivery. It outlines prerequisites including Ably and Confluent Cloud accounts, API keys, Terraform, Node.js, and familiarity with Kafka, React, and Terraform, then directs users to clone a sample project and configure an environment file with an Ably API key. Terraform is used to provision a Confluent Cloud environment, basic Kafka cluster, schema registry, service account, an `orders` topic, and a Datagen source connector that produces mock order events. Users then manually upload the Ably Kafka Connector as a Confluent Cloud custom sink connector, configure it to consume the `orders` topic and publish messages to corresponding Ably channels, allow outbound traffic to Ably’s endpoint, and launch it. Running the included Next.js application displays the generated Kafka messages in a browser over an Ably real-time connection, while final cleanup instructions cover deleting the connector and destroying the Terraform-managed cloud resources to avoid ongoing charges.

Fleet management systems have evolved to collect telemetry data from IoT devices, enabling companies to gain a holistic view of their fleets. However, this has also presented challenges, particularly in handling the ever-increasing amount of data emitted by vehicle IoT sensors in real time. MQTT, a lightweight messaging protocol, is widely used in IoT applications, but it poses limitations such as no stream processing and data reprocessing. To address these challenges, organizations combine MQTT with Apache Kafka, a leading data streaming technology that enables companies to process large volumes of IoT data in real time. Confluent's integration with Kafka facilitates the delivery of a fleet management system that increases efficiency, improves fleet visibility, and provides a better service by enabling real-time analytics and optimized routes.

Wendell Slater, a Cloud Product Representative at Confluent, joined the company nearly three years ago after discovering Kafka through the Streaming Audio podcast and connecting directly with sales managers on LinkedIn. He now helps Confluent Cloud users expand their usage, address early technical questions, and progress toward committed adoption. His team collaborates with Sales, Marketing, and Product on product-led growth initiatives, providing technical support to early-stage prospects alongside SDRs, account executives, and sales engineers. Slater values remote work for its flexibility and uses focused work sessions, walks, and other routines to maintain productivity. He describes Confluent’s culture as empathetic and growth-oriented, highlighting his involvement as a VetNet leader and the company’s eight employee resource groups that support diversity, equity, inclusion, and employee community-building.

Confluent promoted its participation as a Platinum sponsor at AWS re:Invent 2023 in Las Vegas, where it planned to demonstrate Confluent Cloud on AWS and its cloud-native data streaming capabilities. Activities included demonstrations at booth #1000 of the Kora Engine-powered platform for real-time data streaming, processing, governance, and sharing without self-managing Apache Kafka, along with cloud-credit giveaways, refreshments, and a happy hour. The company also highlighted a fireside chat with DISH Network on exchanging data products across systems and businesses using 5G connectivity and data streaming, opportunities to meet Apache Kafka and Apache Flink specialists, and integrations that support pipelines to AWS services such as Amazon S3 and AWS Lambda. Confluent positioned its platform as a way to simplify streaming infrastructure, accelerate product development, reduce total cost of ownership by up to 60%, and make data more accessible for real-time business use cases.

The key points covered in the text relate to the limitations of batch-based risk analysis processes in the past, which often resulted in delayed insights and incorrect data leading to significant financial losses. The author suggests that these issues can be addressed by adopting a data streaming platform approach, which enables real-time processing of large amounts of data, reducing latency and improving accuracy. By utilizing event-driven design and leveraging capabilities like Flink in Confluent Cloud, businesses can create accurate, up-to-date risk aggregation and analysis, simplify their architecture, reduce duplication of data, and save costs. The proposed solution involves creating 'current state' aware data products with in-stream processing, providing continuous updates on real-time exposures, and integrating with downstream analytics and reporting tools to create regulatory risk reports as needed.

Confluent has introduced SQL Workspaces, a browser-based tool for using SQL to interact with data in Confluent Cloud, enhancing the integration of Apache Flink's streaming capabilities with SQL's widespread familiarity. The tool allows users to seamlessly write SQL queries on Kafka data without leaving the Confluent Cloud environment, offering a user-friendly interface that auto-saves SQL statements and supports multiple editors for complex data interactions. SQL Workspaces bridges the gap between batch and real-time data processing by adapting SQL to handle perpetual streaming queries, providing real-time updates, and displaying runtime metrics for ongoing analytics. This integration highlights SQL's enduring relevance in the evolving landscape of data streaming, as Confluent Cloud harmonizes its components to facilitate intuitive data processing and analysis. The platform’s design, including contextual entry points and integration with core Confluent Cloud features, simplifies SQL-based data exploration and empowers organizations to maximize their data's potential.

Confluent Cloud enables financial institutions to exchange data securely, efficiently, and at scale, addressing challenges such as security, onboarding, scaling, multi-tenancy, data formats, chargebacks, and regulatory compliance. By leveraging Confluent Cloud's capabilities, firms can streamline their processes, prevent failures, and automate end-to-end tasks, ultimately driving profitability and defensibility in an increasingly competitive industry. The solution provides features such as streaming governance, data encryption, OAuth, RBAC, client quotas, observability, scaling, and Terraform, which enable financial institutions to implement automation, security, cloud-native, and governance capabilities for their data exchange processes.

The first-ever Confluent Public Sector Summit brought together over 160 participants from diverse backgrounds to discuss harnessing the power of data to empower government missions. The event highlighted several key themes, including trust, AI success dependent on clean and trustworthy data, and the need for training the whole workforce to understand how to use data products. Chief Data Officers (CDOs) shared their experiences in shaping data strategies within their organizations, emphasizing the importance of building trust among stakeholders and creating accessible data platforms. The summit also discussed security, resiliency, and efficiency in data sharing, with a focus on implementing standards and governance to define what data gets shared and with whom. Various panelists showcased innovative solutions using Apache Kafka, highlighting its impact on immediacy in data sharing and mission success.

Financial services firms are increasingly using data streaming to interpret the continuous flow of consumer-generated data and respond to opportunities in real time. Applications include improving customer experiences, strengthening security and governance, modernizing back-end systems, detecting fraud, managing risk, and enabling real-time payments. Confluent promotes its Financial Services Resource Hub as a source of information for organizations developing these capabilities, while related posts highlight the use of open security lakes based on Apache Iceberg and the importance of sustained value and execution in building customer loyalty.

Dialpad, a customer intelligence platform provider, modernized its data infrastructure to support real-time analytics and user-facing features. The company required four core capabilities in a replacement system: focus on streaming data, support for upserts, flexible indexing capabilities, and an open-source core backed by a strong sponsoring company. Dialpad turned to Confluent Cloud for fast and scalable data streaming and StarTree Cloud, powered by Apache Pinot, which provided the necessary scalability and speed for real-time analytics. The new architecture has reduced average end-to-end ingestion latencies from over one minute down to an average of 500 ms, greatly improving data freshness and enabling Dialpad to create new products and features that rely on real-time metrics.