August 2023 Summaries

This blog post discusses how to tune your source connectors in Kafka Connect to improve throughput by increasing the number of records pulled from the endpoint and reducing the time it takes to apply transformations. The authors walk through an example using a JDBC Source connector, explaining that the Converter and Transformations stages are not tunable, but the Connector and Producer configurations can be modified to optimize performance. They provide a step-by-step guide on how to determine which configurations to change first, starting with producer-level metrics such as batch-size-avg and records-per-request-avg, and then tuning connector-level metrics like poll.interval.ms to improve throughput. The authors demonstrate an example where the throughput is increased by 57% from 50.9 MB/s to 88.2 MB/s by tuning both the producer and connector configurations. They emphasize that this methodology can be applied to any source connector and that understanding how a connector works is essential for effective tuning.

The text details a case study of an organization that enhanced its mobile application to provide personalized content using Confluent Cloud, in conjunction with Apache Kafka, Apache Flink, and MongoDB. It begins by highlighting the challenge of delivering standout notifications to smartphone users, who frequently check their devices. The organization in question sought to capture a significant portion of the smartphone market, necessitating the handling of vast and diverse data sets to deliver real-time, relevant content to users across the globe. Initially, their infrastructure struggled with scalability and efficiency, especially during high-traffic events, and faced challenges in real-time data delivery and machine learning model training. By transitioning to Confluent Cloud's managed data streaming platform, the organization overcame these hurdles through improved scalability, real-time processing, and cost-effective operations. Key features like fanout capabilities, stream processing, and extensive integration support enabled the organization to improve content personalization, reduce operational downtime, and focus on new business initiatives. The implementation of Confluent Cloud facilitated seamless integration and expansion of their capabilities, enabling them to handle increased data volumes and deliver enhanced user experiences, ultimately reducing customer churn and fostering business growth.

The Very Large Data Bases (VLDB) conference is a premier event for data management systems, and Confluent's paper on "Kora: A Cloud-Native Event Streaming Platform For Kafka" was awarded the Best Industry Paper at VLDB 2023. Kora is a next-generation engine that serves the Kafka protocol, designed to provide reliability, elasticity, and cost-efficiency in cloud environments. The system allows users to think in terms of their workload requirements rather than infrastructure, enabling consistent performance across diverse cloud capabilities. Confluent's CEO Jay Kreps wrote about why they built Kora, highlighting market demand for a cloud-native Kafka service that captures the benefits of cloud computing. The development of Kora was a challenging but transformative project that included breakthroughs in cellular architecture, serverless abstraction, and self-balancing clusters. The system has been powering 30,000 Confluent Cloud clusters and thousands of customers, offering better scalability, reliability, performance, and cost efficiency.

Confluent Platform 7.5 introduces several enhancements aimed at improving security, developer efficiency, and disaster recovery. The update includes single sign-on (SSO) for Control Center, allowing seamless integration with identity providers like OpenID Connect for improved security compliance and user management. The REST Proxy Produce API v3 simplifies data access and integration, enabling users to interact with the Confluent Platform through REST calls in their preferred programming languages without deep knowledge of Kafka. Furthermore, the platform enhances disaster recovery capabilities through bidirectional Cluster Linking, allowing active-active and active-passive architectures with efficient data replication and consumer offset management. Additional features include FIPS support for enhanced security compliance and improvements in deploying and managing connectors using Confluent for Kubernetes. Built on Apache Kafka 3.5, the platform offers a comprehensive solution for handling data in motion with cloud-native capabilities.

The text discusses the use of Apache Flink® for stream processing in various industries and applications. It highlights three main categories of use cases: event-driven applications, real-time analytics, and streaming data pipelines. Event-driven applications are used in finance, healthcare, and transportation to automate responses to events as they happen. Real-time analytics involves analyzing real-time data streams to generate important business insights for decision-making. Streaming data pipelines continuously ingest data streams from various sources, perform joins, aggregations, and transformations to create new enriched streams of higher value. The text also mentions how Flink can be used with Kafka as a complete data streaming platform.

Confluent has achieved the Google Cloud Ready - Cloud SQL Designation, marking a significant milestone in its collaboration with Google Cloud to optimize data integration solutions for Cloud SQL, Google Cloud's managed relational database service. This designation highlights Confluent's ability to integrate its solutions with Cloud SQL, ensuring consistent, real-time data streaming that addresses the challenges faced by businesses due to disparate data systems. By enabling seamless data flow across various environments, Confluent and Google aim to help organizations modernize their infrastructures, improve customer experiences, and harness the power of real-time data analytics. This partnership supports businesses in overcoming the inefficiencies of legacy systems and facilitates a more agile approach to data management, ultimately empowering enterprises to make informed decisions and drive innovation.

Confluent has expanded its partnership with Google Cloud, emphasizing increased technical support and integration with Google Cloud’s data, analytics, and AI services to enhance their joint offerings. They have been recognized as a Google Cloud Technology Partner of the Year for Marketplace–Data and Analytics, highlighting the significance of cloud marketplaces in meeting customer needs. The collaboration aims to solve complex data-streaming challenges for companies by leveraging Confluent's data streaming platform alongside Google Cloud's capabilities, facilitating real-time data processing and personalized consumer experiences. Confluent enables organizations to stream data into Google Cloud services like BigQuery and Vertex AI, enhancing machine learning and analytics applications. Their cloud-native platform, powered by the Kora engine, offers significantly improved scalability and performance over Apache Kafka while reducing costs. Confluent's alignment with Google Cloud has led to achievements such as Google Cloud Ready validations for services like BigQuery and CloudSQL and the release of new connectors to support analytical use cases. The partnership aims to help organizations modernize data platforms and bridge on-premises and multi-cloud architectures through a suite of enhanced services and marketplace options.

The blog post discusses the introduction of Versioned State Stores in Kafka Streams with the release of Apache Kafka 3.5, highlighting how they address issues related to out-of-order data processing by storing multiple record versions per key. These stores provide timestamped lookups, allowing applications to handle data with varying timestamps more accurately, as illustrated through examples like a restaurant billing system and voting aggregation. By shifting from latest-by-offset to latest-by-timestamp semantics, Versioned State Stores enhance the handling of table aggregations and joins, ensuring that the most accurate and timely data is used in calculations. The article details the implementation challenges and considerations, such as performance impacts due to the need for additional storage and processing resources, and provides guidance on upgrading to versioned stores. The post also notes that while versioned stores are currently opt-in and lack interactive query support, further developments are anticipated to enhance their functionality.

Real-time streaming data is crucial for businesses to harness valuable insights and maintain a competitive edge in a fast-paced environment. This technology allows for the immediate processing and analysis of data, enhancing decision-making, reducing latency, and improving AI and ML responsiveness. Streaming data is unbounded, unordered, and immutable, coming from various sources, and is essential for applications like fraud detection, inventory management, and customer personalization. Despite its importance, many organizations face challenges in adopting streaming architectures due to the complexity and expertise required. Managed services, like Confluent Cloud, offer solutions by simplifying the deployment and management of streaming platforms, ensuring scalability, reliability, and compliance. These services enable businesses to easily integrate streaming data into their operations, thereby enhancing agility, reducing costs, and delivering better customer experiences. As data streaming becomes the norm, companies that fail to adopt these technologies risk falling behind in the digital transformation race.

Apache Flink has gained significant popularity due to its unified stream and batch processing framework, which provides high resource efficiency, massive throughput with low latency, and robust state handling. The platform's flexibility allows it to be tailored to a wide range of workloads, including streaming and batch workloads. Apache Flink offers four different APIs, each tailored to different users and use cases, providing fine-grained control over data processing. The platform unifies stream and batch processing, allowing for consistent semantics across real-time and historical data processing use cases. Additionally, Flink has been hardened in production with various features such as a flexible metrics system, extensive observability, and savepoints. Its integration with Apache Kafka provides a seamless experience for stream processing, leveraging the latter's core streaming data storage layer. Many leading innovators have adopted both Kafka and Flink in their data stack, making it an attractive choice for organizations looking to build powerful data streaming architectures.

Real-time AI applications are becoming increasingly essential across various industries, necessitating access to up-to-date data to provide accurate and responsive user experiences. Confluent and Rockset together form a powerful architecture for enabling real-time AI by combining Confluent's data streaming capabilities with Rockset's vector search functionality. This combination is crucial for applications like Whatnot's live auction platform, which relies on real-time data to recommend live streams effectively. Confluent Cloud provides a comprehensive data streaming solution that integrates seamlessly with various systems, while Rockset offers low-latency, high-concurrency query capabilities, making it ideal for real-time AI applications. Whatnot's use of this technology stack has significantly improved their recommendation engine, allowing for personalized suggestions in real-time and supporting their rapid growth. The synergy between Confluent and Rockset exemplifies how businesses can leverage real-time data to enhance AI-driven applications efficiently and at scale.

Apache Kafka is a distributed computing system that enables multiple servers to work together for speed, durability, and scalability. It uses a cluster structure, where brokers manage load balancing, replication, and stream decoupling, and partitions are used to balance the workload across the cluster. Data is replicated across multiple servers to ensure stability and availability. Kafka balances load through load balancing and partitioning, allowing it to handle large volumes of data and provide low latency. The system uses a leader-follower model for replication, where followers replicate writes from the leader node to maintain data consistency. In case of failure, KRaft, a metadata management system, is used to manage the cluster's metadata and trigger leader elections. Kafka can be run on-premise or in the cloud, providing developers with flexibility in choosing their deployment option. The system also uses events as a fundamental concept, enabling event-driven design, event sourcing, designing events, and event streaming.

The Confluent maturity curve outlines five levels of adoption for data streaming technologies like Apache Kafka. The first three levels (1-3) are characterized by bottom-up, tech-led adoption, while the fourth level (4) requires top-down, business-led adoption. Level 4 involves coordinating disparate teams, implementing a Center of Excellence, and managing data as a product to achieve economies of scale. To move from Level 3 to 4, organizations need education and awareness from the business, budget or solid business case outlining real benefits, senior-level sponsorship, and a change management approach. The Confluent maturity curve is being revised to emphasize the step from Level 3 to 4, with recommendations including writing a strategy or internal proposal, determining metrics and ROI, and implementing a people, process, and technology layer to support this shift. Organizations can benefit from managing data like a product, driving innovation, cost allocation, and risk management, while also reducing costs associated with managing data. Change management is critical for this transformation, involving identifying stakeholders, training, education sessions, tech talks, and new performance metrics.

The text discusses the integration of AWS Lambda with Kafka topics to create scalable, event-driven architectures suitable for various industries, highlighting two primary integration patterns: using AWS Lambda as a sink connector and configuring Kafka as an event source. The Lambda Sink Connector allows synchronous or asynchronous invocation of functions, offering flexibility in workload management, while the event source mapping pattern automatically invokes Lambda functions based on message flow, supporting auto-scaling and load monitoring. The text provides detailed insights into each pattern's operational features, such as message processing, error handling with dead-letter queues, and parallel task executions. It also presents real-world use cases in healthcare and email marketing, illustrating the practical applications of these integrations. Additionally, the text mentions Confluent Cloud's support for multiple Lambda functions and cost-effective scaling, with an example from BT Group's use of Confluent for real-time data streaming in telecom.