October 2023 Summaries

The text discusses the challenges and solutions in implementing AI and ML-driven business analytics, emphasizing the importance of data quality, performance, and scalability. It highlights the role of AI and ML in predicting behaviors, discovering trends, and uncovering hidden relationships, but points out the difficulties companies face with data integrity, context, and consistency, mainly due to human and organizational factors. The text introduces Confluent Cloud as a comprehensive real-time data platform that addresses these challenges by offering tools for data aggregation, stream processing, and governance, which help maintain data integrity and quality. It explains how Confluent Cloud's capabilities, such as fully managed Kafka Connectors and stream processing tools, enable businesses to consolidate data from various sources and perform real-time analysis, making it a preferred choice for handling large-scale AI and ML workloads. The text also provides industry-specific examples, illustrating how Confluent's platform can be used for predictive analytics in healthcare, fraud prevention in finance, inventory management in retail, customer retention, and maintenance optimization.

Apache Flink 1.18.0 introduces a multitude of enhancements, including a JDBC Driver for the Flink SQL Gateway, allowing users to interact with tables via SQL clients that support JDBC. The update also facilitates the use of stored procedures within Flink connectors, enabling easier and potentially optimized data operations. New commands such as REPLACE TABLE AS SELECT now support atomicity, while TRUNCATE TABLE is available in batch mode. Time travel SQL syntax is supported for querying historical data versions, and operator-level state time-to-live (TTL) settings are introduced for stream regular joins. Additional improvements include Hybrid Shuffle for remote storage, runtime filters for optimized join performance, and the Adaptive Scheduler for better elasticity and resource management. Flink now runs on Java 17, and enhancements in SQL client usability and batch processing performance have been implemented. The update marks a shift from Akka to Apache Pekko due to licensing changes and includes upgrades to Apache Calcite, enhancing Flink's query optimization capabilities. The community continues to focus on cloud-native integration and prepares for the upcoming Flink 2.0 release by deprecating certain legacy APIs.

This series of blog posts aims to guide users from absolute beginners to building a fully functioning, scalable Generative AI application using Kappa Architecture and the Kora engine. The first post delves into the basics of GenAI and large language models (LLMs), explaining how they work and how to integrate them within an organization. It also covers different ways to integrate generative AI into a business, including accessing tools like ChatGPT or Bard, integrating aspects of applications with third-party LLMs via APIs, using open source LLMs, and training custom models from scratch. The subsequent posts will explore the Kappa Architecture, context, memory, private data, scaling up, and real-world production application integration with GPT-4 and streaming data.

NASA's innovative use of Apache Kafka® for data streaming has led to significant advancements in its operations, earning the agency the Data Streaming Company of the Year award in 2022. At the Current 2023: The Next Generation of Kafka Summit, Joe Foster, NASA's Cloud Computing Program Manager, discussed the agency's embrace of cloud computing and data streaming with Confluent CEO Jay Kreps. A notable project is the General Coordinates Network (GCN), a platform for real-time astronomical alerts that exemplifies the agency's commitment to crowdsourcing scientific observations. Another project, DAPHNE, demonstrates NASA's shift towards cloud-based standardized hardware, significantly cutting costs. These advancements are part of NASA's broader strategy to overcome the challenges of its distributed organizational structure and enhance real-time data streaming capabilities, crucial for timely scientific observations like supernovas. Foster's role has been pivotal in accelerating cloud adoption across NASA, with his team developing the Mission Cloud Platform to facilitate this transition. By focusing on innovation rather than mere data migration, NASA aims to democratize data through initiatives like the Data and Reasoning Fabric, preparing the agency for future technological challenges and opportunities.

Service NSW, an Australian government agency, aims to become the world's most customer-centric government by 2030. To achieve this, they have developed a single view of the customer (SVOC) using event-driven architecture and Confluent's streaming data platform. This program consolidates and organizes customer information from various sources, providing a centralized location for agency users to access. The SVOC enables targeted personalization, real-time support, and improved customer experiences by leveraging enriched customer data for future products and services. With Confluent's platform, Service NSW can handle any type of infrastructure and data source, ensuring secure and compliant management of customer information.

The Confluent Schema Registry is crucial for effective communication between producers and consumers in a streaming platform by ensuring consistent schema usage, even as schemas evolve. As streaming becomes more prevalent, organizations are shifting responsibilities, such as data quality and transformation, to the data source, a process known as "shift-left," which emphasizes the use of data contracts. These contracts serve as formal agreements on data structure and semantics between upstream and downstream components, enhancing data transparency, reliability, and quality. The article provides an example of evolving a schema into a comprehensive data contract, illustrating steps like adding business metadata, data quality rules, and custom actions. It also discusses handling backward-incompatible schema changes using compatibility groups and migration rules with JSONata, highlighting the power of combining these elements for seamless schema evolution. The focus on data contracts allows for greater reliability in streaming, supported by the Confluent Schema Registry, which makes streaming more robust and dependable.

Businesses, especially in the retail sector, are increasingly reliant on enriched, real-time data from multiple sources to meet modern customer expectations for convenience and informed purchasing decisions. Traditional methods of managing retail inventory using batch processing and outdated data have proven inadequate, leading to customer dissatisfaction and lost sales. Confluent's data streaming platform offers a solution by enabling near real-time data consumption and processing from various sources, creating a continuously updated stream of inventory data that can be used by commerce, replenishment, allocation, and merchandising systems. This approach not only enhances customer experience by providing accurate and timely inventory information but also optimizes operations, reducing complexity, costs, and improving scalability. Real-time inventory management ensures better revenue capture and demand estimation and supports automation in replenishment systems, contributing to a more resilient and efficient supply chain.

Real-time data streaming and processing technologies are transforming the way businesses handle data, shifting from a traditional "store then process" approach to a "process then store" paradigm. This evolution necessitates understanding new concepts and evaluating key platforms like Apache Flink and Kafka Streams, which are crucial for stream processing due to their capabilities in handling large-scale, real-time data tasks. Apache Flink, known for its rich API and lower latency, is ideal for complex event processing and machine learning, while Kafka Streams offers simplicity and tight integration with Kafka. Building a robust streaming framework requires understanding the vast landscape of streaming solutions, which include data streaming platforms, stream processing engines, and managed services that simplify the complexities of deployment and operation. Managed services, such as Confluent, provide automated infrastructure management and reduce operational burdens, offering scalability and resilience while allowing businesses to focus on application development. The decision between managed and self-managed solutions depends on factors such as deployment complexity, data residency, and regulatory requirements. Ultimately, businesses must carefully consider their specific streaming needs to maximize the return on investment and leverage the benefits of real-time data integration and analytics.

Apache Kafka 3.6.0 has been released, bringing several new features and improvements, including Tiered Storage, which provides a separation of computation and storage in the broker for pluggable storage tiering. This feature is currently an early access release suitable only for testing in non-production environments. Additionally, this release introduces improved durability for the KRaft layer by adding metadata transactions that consist of BeginTransaction, Number of records, EndTransaction or AbortTransaction. The release also includes support for Delegation Tokens in KRaft, further reducing the gap between ZooKeeper-based Kafka clusters and KRaft. Furthermore, this release upgrades the ZooKeeper version bundled with Kafka to 3.8.2, improves data integrity by adding more validation logic for message timestamps, and introduces new metrics for KRaft performance. Other notable improvements include support for topic-mutating SMTs for async sink connectors, first-class admin support for offsets in Kafka Connect, and enhanced plugin class discovery for Connect workers.

Organizations today face multifaceted security challenges requiring collaboration across departments, similar to the teamwork displayed by Dorothy and her friends in "The Wizard of Oz." However, security operations often remain siloed, with different teams and tools failing to integrate, which impedes comprehensive threat response. A shift towards data-centric security is proposed, where data is treated as a strategic asset accessible across the organization. This approach involves modernizing Security Incident and Event Management (SIEM) and Security Orchestration, Automation, and Response (SOAR) tools, which traditionally aggregate data from various sources but struggle with growing data volumes and integration. Data streaming is championed as a solution to pre-process and curate data before it reaches SIEM or SOAR systems, reducing costs and workload. An upcoming webinar aims to demonstrate how data streaming can enhance security operations by integrating upstream data processing, with government examples illustrating its effectiveness in improving efficiency and proactivity in security measures.

Confluent Cloud provides a fully managed cloud-native data streaming platform that enables organizations to harness the power of data to predict future customer demands accurately, optimize operational efficiency, and drive strategic growth. By leveraging Confluent Cloud, businesses can scale their operations based on fluctuating data demands, manage infrastructure management tasks, and ensure minimal downtime and uninterrupted data streaming. This is achieved through a curated data ecosystem that combines Confluent with Elasticsearch, MongoDB, Snowflake, and Snowpipe to provide a comprehensive data pipeline from data ingestion to analysis, enabling organizations to make data-driven decisions and gain insights from various types of data in real-time scenarios. The platform has been successfully adopted by Acme Inc., which uses cutting-edge technology to gather data from diverse sources, enriches the raw data with context, and provides clients with a deeper understanding of its potential impact on their businesses.

Artificial intelligence is significantly enhancing social media by personalizing content, moderating inappropriate material, managing brand reputation, and analyzing sentiment, among other applications. As social media's influence continues to grow, AI-driven solutions, such as those provided by data streaming platforms like Confluent, are being leveraged to monitor and maintain brand reputation in real-time. Confluent, built on Apache Kafka, offers capabilities such as pre-built connectors, stream processing, and real-time data ingestion, enabling organizations to efficiently manage and analyze vast amounts of social media data. By implementing a real-time brand reputation monitoring system, PR teams can quickly identify and address potentially damaging posts across various platforms, reducing the risk of reputational harm. Confluent's architecture supports this process through its scalability, managed connectors, and event-driven data flow, positioning it as a crucial tool for modern reputation management in the dynamic landscape of social media.

The text discusses the importance of personalization in retail, particularly in hybrid shopping experiences where customers expect a seamless experience between online and offline interactions. To achieve this, companies need to collect and analyze data from various sources, including customer interactions, inventory levels, and location data. The article highlights the challenges of integrating data from disparate systems and the need for real-time integration platforms that can handle high volumes of data and provide insights for personalized marketing efforts. It also discusses the role of stream processing and message delivery in enabling real-time personalization, and provides a technical quickstart example using Confluent Cloud and Ably to demonstrate how to build a hybrid in-store personalized experience. The article concludes by emphasizing the benefits of personalization for retail organizations, including improved customer satisfaction, increased conversion rates, and enhanced employee engagement.

Interoperability between healthcare systems is crucial for delivering quality services to patients and providers, as it enables the secure exchange of health information across separate entities. However, many healthcare payers face technical challenges in implementing interoperability due to siloed data and legacy communication methods, which can lead to higher administrative costs and limited member experiences. To address these challenges, data streaming technology such as Apache Kafka and Confluent has been used to decouple proprietary monolithic applications and unlock real-time data for use in event-driven microservices. By leveraging data streaming, healthcare payers can improve care coordination, increase efficiency, better analyze data, prevent fraud and abuse, and provide a better member experience, ultimately delivering better patient outcomes.

Confluent Cloud is being used by DISH Wireless to build a programmable network that decouples data ownership, implements federated governance, democratizes access to trustworthy data products through telco APIs, and accelerates insights and innovation for customers and their developers. The company has adopted Confluent Cloud's cloud-native data streaming capabilities and focused on building a 5G network using data mesh principles without taking on the operational burden of self-managing open-source Kafka. DISH Wireless has built a wireless connectivity platform from the ground up, combining best-in-class components to create an open-RAN (O-RAN) network that allows developers and organizations to combine their own network elements. The company is utilizing Confluent Cloud's cloud-native capabilities and enterprise-grade tooling to create trustworthy APIs that developers can leverage for customized connectivity. With Confluent Cloud at the core of its connectivity platform, DISH Wireless is able to build streaming pipelines that make its 5G network data instantly accessible to developers, unlocking connected data streaming use cases in retail, manufacturing, and automotive industries.