November 2023 Summaries

This blog post discusses how businesses can leverage real-time data streaming and processing technologies to drive value using AI/ML models. It highlights the importance of streaming analytics infused with AI and ML in various domains, such as fraud prevention, autonomous vehicles, voice assistants, recommendation engines, dynamic pricing, predictive maintenance, and intelligent cybersecurity systems. The post also emphasizes the role of data streaming platforms in powering modern AI/ML models with real-time, trusted data across the enterprise. It further explains how Confluent Cloud simplifies the integration of streaming data into AI/ML models by abstracting away technical complexity and supporting various features like model deployment, monitoring, and embedding models into a real-time stream. The post concludes that real-time AI/ML is becoming a strategic differentiator as it increasingly becomes important to incorporate real-time context from streaming data into the decision-making process.

The financial industry is evolving to adopt modern core banking systems that can respond and scale to meet customer demand in near-real time. Traditional mainframe-based architectures are being replaced by event-driven streaming data platforms, which enable banks to unlock their critical mainframe data, minimize disruptions, and power their ever-growing landscape of cloud-native applications and systems. The key technologies used in the modern core banking stack include unlocking core banking system of record, enriching data in real-time with Apache Flink, decoupling system of record and system of access with MongoDB, and writing data out to a non-relational DB like Mongo Atlas. Mainframe migrations are being done incrementally, often through connectors like MQ on Z and Oracle CDC, which reduce costs by 90% or more, depending on the solution. The modernization process is slow, iterative, and typically not big bang, with many large financial institutions still in the early innings of this modernization.

Navigating the complexities of Apache Kafka can be challenging, but a variety of resources from Confluent can facilitate a smoother learning experience. With a rich library of educational content, beginners can access tutorials on building Kafka producer and consumer applications, as well as understanding core concepts such as partitioning, replication, and Kafka Streams. Notable resources include Tim Berglund's YouTube series, which breaks down Kafka fundamentals and provides hands-on exercises, and the Kafka Streams 101 course, which covers advanced topics like handling out-of-order data. Tutorials offer practical, step-by-step guidance for building applications in cloud-based or standalone environments, accommodating various programming languages such as Java, JavaScript, Python, and .NET/C#. Additionally, community support is available through Slack and forums, helping learners overcome challenges and further explore Kafka's versatility in event-driven design and data streaming.

Fleet management systems have evolved to collect telemetry data from IoT devices, enabling companies to gain a holistic view of their fleets. However, this has also presented challenges, particularly in handling the ever-increasing amount of data emitted by vehicle IoT sensors in real time. MQTT, a lightweight messaging protocol, is widely used in IoT applications, but it poses limitations such as no stream processing and data reprocessing. To address these challenges, organizations combine MQTT with Apache Kafka, a leading data streaming technology that enables companies to process large volumes of IoT data in real time. Confluent's integration with Kafka facilitates the delivery of a fleet management system that increases efficiency, improves fleet visibility, and provides a better service by enabling real-time analytics and optimized routes.

The key points covered in the text relate to the limitations of batch-based risk analysis processes in the past, which often resulted in delayed insights and incorrect data leading to significant financial losses. The author suggests that these issues can be addressed by adopting a data streaming platform approach, which enables real-time processing of large amounts of data, reducing latency and improving accuracy. By utilizing event-driven design and leveraging capabilities like Flink in Confluent Cloud, businesses can create accurate, up-to-date risk aggregation and analysis, simplify their architecture, reduce duplication of data, and save costs. The proposed solution involves creating 'current state' aware data products with in-stream processing, providing continuous updates on real-time exposures, and integrating with downstream analytics and reporting tools to create regulatory risk reports as needed.

Confluent has introduced SQL Workspaces, a browser-based tool for using SQL to interact with data in Confluent Cloud, enhancing the integration of Apache Flink's streaming capabilities with SQL's widespread familiarity. The tool allows users to seamlessly write SQL queries on Kafka data without leaving the Confluent Cloud environment, offering a user-friendly interface that auto-saves SQL statements and supports multiple editors for complex data interactions. SQL Workspaces bridges the gap between batch and real-time data processing by adapting SQL to handle perpetual streaming queries, providing real-time updates, and displaying runtime metrics for ongoing analytics. This integration highlights SQL's enduring relevance in the evolving landscape of data streaming, as Confluent Cloud harmonizes its components to facilitate intuitive data processing and analysis. The platform’s design, including contextual entry points and integration with core Confluent Cloud features, simplifies SQL-based data exploration and empowers organizations to maximize their data's potential.

Confluent Cloud enables financial institutions to exchange data securely, efficiently, and at scale, addressing challenges such as security, onboarding, scaling, multi-tenancy, data formats, chargebacks, and regulatory compliance. By leveraging Confluent Cloud's capabilities, firms can streamline their processes, prevent failures, and automate end-to-end tasks, ultimately driving profitability and defensibility in an increasingly competitive industry. The solution provides features such as streaming governance, data encryption, OAuth, RBAC, client quotas, observability, scaling, and Terraform, which enable financial institutions to implement automation, security, cloud-native, and governance capabilities for their data exchange processes.

The first-ever Confluent Public Sector Summit brought together over 160 participants from diverse backgrounds to discuss harnessing the power of data to empower government missions. The event highlighted several key themes, including trust, AI success dependent on clean and trustworthy data, and the need for training the whole workforce to understand how to use data products. Chief Data Officers (CDOs) shared their experiences in shaping data strategies within their organizations, emphasizing the importance of building trust among stakeholders and creating accessible data platforms. The summit also discussed security, resiliency, and efficiency in data sharing, with a focus on implementing standards and governance to define what data gets shared and with whom. Various panelists showcased innovative solutions using Apache Kafka, highlighting its impact on immediacy in data sharing and mission success.

Dialpad, a customer intelligence platform provider, modernized its data infrastructure to support real-time analytics and user-facing features. The company required four core capabilities in a replacement system: focus on streaming data, support for upserts, flexible indexing capabilities, and an open-source core backed by a strong sponsoring company. Dialpad turned to Confluent Cloud for fast and scalable data streaming and StarTree Cloud, powered by Apache Pinot, which provided the necessary scalability and speed for real-time analytics. The new architecture has reduced average end-to-end ingestion latencies from over one minute down to an average of 500 ms, greatly improving data freshness and enabling Dialpad to create new products and features that rely on real-time metrics.