August 2023 Summaries

This blog post discusses how to tune your source connectors in Kafka Connect to improve throughput by increasing the number of records pulled from the endpoint and reducing the time it takes to apply transformations. The authors walk through an example using a JDBC Source connector, explaining that the Converter and Transformations stages are not tunable, but the Connector and Producer configurations can be modified to optimize performance. They provide a step-by-step guide on how to determine which configurations to change first, starting with producer-level metrics such as batch-size-avg and records-per-request-avg, and then tuning connector-level metrics like poll.interval.ms to improve throughput. The authors demonstrate an example where the throughput is increased by 57% from 50.9 MB/s to 88.2 MB/s by tuning both the producer and connector configurations. They emphasize that this methodology can be applied to any source connector and that understanding how a connector works is essential for effective tuning.

The text details a case study of an organization that enhanced its mobile application to provide personalized content using Confluent Cloud, in conjunction with Apache Kafka, Apache Flink, and MongoDB. It begins by highlighting the challenge of delivering standout notifications to smartphone users, who frequently check their devices. The organization in question sought to capture a significant portion of the smartphone market, necessitating the handling of vast and diverse data sets to deliver real-time, relevant content to users across the globe. Initially, their infrastructure struggled with scalability and efficiency, especially during high-traffic events, and faced challenges in real-time data delivery and machine learning model training. By transitioning to Confluent Cloud's managed data streaming platform, the organization overcame these hurdles through improved scalability, real-time processing, and cost-effective operations. Key features like fanout capabilities, stream processing, and extensive integration support enabled the organization to improve content personalization, reduce operational downtime, and focus on new business initiatives. The implementation of Confluent Cloud facilitated seamless integration and expansion of their capabilities, enabling them to handle increased data volumes and deliver enhanced user experiences, ultimately reducing customer churn and fostering business growth.

The Very Large Data Bases (VLDB) conference is a premier event for data management systems, and Confluent's paper on "Kora: A Cloud-Native Event Streaming Platform For Kafka" was awarded the Best Industry Paper at VLDB 2023. Kora is a next-generation engine that serves the Kafka protocol, designed to provide reliability, elasticity, and cost-efficiency in cloud environments. The system allows users to think in terms of their workload requirements rather than infrastructure, enabling consistent performance across diverse cloud capabilities. Confluent's CEO Jay Kreps wrote about why they built Kora, highlighting market demand for a cloud-native Kafka service that captures the benefits of cloud computing. The development of Kora was a challenging but transformative project that included breakthroughs in cellular architecture, serverless abstraction, and self-balancing clusters. The system has been powering 30,000 Confluent Cloud clusters and thousands of customers, offering better scalability, reliability, performance, and cost efficiency.

Current 2023: The Next Generation of Kafka Summit, held September 26–27 in San Jose, was presented as a two-day data streaming conference featuring more than 100 sessions, networking, technical training, certifications, and community events. Its keynotes were expected to cover the evolution of data streaming platforms, Confluent product developments, customer use cases, generative AI applications, and open-source Kafka and Flink work. Recommended sessions addressed applying data mesh principles to operational and analytical data, real-time machine learning, Flink SQL challenges such as late data and change events, declarative streaming pipelines using Flink and Kubernetes, and Robinhood’s Kafka consumer proxy for separating infrastructure concerns from application logic. Additional programming included informal technical meetups, industry lunches, an AWS GameDay workshop, diversity and inclusion events, and a virtual Streaming Pass for remote attendees.

Confluent Platform 7.5 introduces several enhancements aimed at improving security, developer efficiency, and disaster recovery. The update includes single sign-on (SSO) for Control Center, allowing seamless integration with identity providers like OpenID Connect for improved security compliance and user management. The REST Proxy Produce API v3 simplifies data access and integration, enabling users to interact with the Confluent Platform through REST calls in their preferred programming languages without deep knowledge of Kafka. Furthermore, the platform enhances disaster recovery capabilities through bidirectional Cluster Linking, allowing active-active and active-passive architectures with efficient data replication and consumer offset management. Additional features include FIPS support for enhanced security compliance and improvements in deploying and managing connectors using Confluent for Kubernetes. Built on Apache Kafka 3.5, the platform offers a comprehensive solution for handling data in motion with cloud-native capabilities.

The text discusses the use of Apache Flink® for stream processing in various industries and applications. It highlights three main categories of use cases: event-driven applications, real-time analytics, and streaming data pipelines. Event-driven applications are used in finance, healthcare, and transportation to automate responses to events as they happen. Real-time analytics involves analyzing real-time data streams to generate important business insights for decision-making. Streaming data pipelines continuously ingest data streams from various sources, perform joins, aggregations, and transformations to create new enriched streams of higher value. The text also mentions how Flink can be used with Kafka as a complete data streaming platform.

Confluent has achieved the Google Cloud Ready - Cloud SQL Designation, marking a significant milestone in its collaboration with Google Cloud to optimize data integration solutions for Cloud SQL, Google Cloud's managed relational database service. This designation highlights Confluent's ability to integrate its solutions with Cloud SQL, ensuring consistent, real-time data streaming that addresses the challenges faced by businesses due to disparate data systems. By enabling seamless data flow across various environments, Confluent and Google aim to help organizations modernize their infrastructures, improve customer experiences, and harness the power of real-time data analytics. This partnership supports businesses in overcoming the inefficiencies of legacy systems and facilitates a more agile approach to data management, ultimately empowering enterprises to make informed decisions and drive innovation.

Confluent has expanded its partnership with Google Cloud, emphasizing increased technical support and integration with Google Cloud’s data, analytics, and AI services to enhance their joint offerings. They have been recognized as a Google Cloud Technology Partner of the Year for Marketplace–Data and Analytics, highlighting the significance of cloud marketplaces in meeting customer needs. The collaboration aims to solve complex data-streaming challenges for companies by leveraging Confluent's data streaming platform alongside Google Cloud's capabilities, facilitating real-time data processing and personalized consumer experiences. Confluent enables organizations to stream data into Google Cloud services like BigQuery and Vertex AI, enhancing machine learning and analytics applications. Their cloud-native platform, powered by the Kora engine, offers significantly improved scalability and performance over Apache Kafka while reducing costs. Confluent's alignment with Google Cloud has led to achievements such as Google Cloud Ready validations for services like BigQuery and CloudSQL and the release of new connectors to support analytical use cases. The partnership aims to help organizations modernize data platforms and bridge on-premises and multi-cloud architectures through a suite of enhanced services and marketplace options.

Yash Mayya, a senior software engineer at Confluent, shifted from an early ambition to become a pilot after discovering programming at age nine and joined the company after earning a computer science degree. On the Kafka Connect team, he helps develop a distributed data-integration framework that moves data between external systems and Apache Kafka, with current work focused on expanding Kafka Connect capabilities for Confluent Cloud. His responsibilities include contributing to the Apache Kafka open-source project and addressing cloud networking challenges alongside colleagues in India, the United States, and the wider open-source community. Mayya describes remote collaboration across time zones as occasionally challenging but supported by considerate scheduling, regular team meetings, and in-person gatherings. He cites Confluent’s complex technical work, open-source focus, flexible culture, and emphasis on being smart, humble, and empathetic as reasons for joining and remaining at the company, where he sees continued opportunities for mentorship and career development.

The blog post discusses the introduction of Versioned State Stores in Kafka Streams with the release of Apache Kafka 3.5, highlighting how they address issues related to out-of-order data processing by storing multiple record versions per key. These stores provide timestamped lookups, allowing applications to handle data with varying timestamps more accurately, as illustrated through examples like a restaurant billing system and voting aggregation. By shifting from latest-by-offset to latest-by-timestamp semantics, Versioned State Stores enhance the handling of table aggregations and joins, ensuring that the most accurate and timely data is used in calculations. The article details the implementation challenges and considerations, such as performance impacts due to the need for additional storage and processing resources, and provides guidance on upgrading to versioned stores. The post also notes that while versioned stores are currently opt-in and lack interactive query support, further developments are anticipated to enhance their functionality.

Powerledger, an Australian blockchain-based energy platform founded in 2016, aims to democratize electricity markets by enabling peer-to-peer renewable energy trading and building toward a global carbon-free energy marketplace. It views decentralization, decarbonization, democratization, digitalization, and real-time data as major forces reshaping an industry historically based on one-way energy distribution, with potential benefits including greater renewable integration, traceability, transparency, consumer choice, and lower costs. To support plans to connect 100 million smart meters and serve one billion users within five years, Powerledger adopted Confluent’s managed data-streaming services in 2022 alongside MongoDB Atlas and its existing Apache Kafka infrastructure. The company reports substantial gains in scalability and speed, including a new blockchain capable of 60,000 transactions per second compared with 12 on its prior Ethereum-based system, processing one million records in seven minutes, and uploading data from 25,000 meters within minutes rather than hours. Powerledger also says the platform has reduced data-processing burdens on developers, improved feature delivery, and provided the reliability and security needed for an essential, regulated energy service.

Real-time streaming data is crucial for businesses to harness valuable insights and maintain a competitive edge in a fast-paced environment. This technology allows for the immediate processing and analysis of data, enhancing decision-making, reducing latency, and improving AI and ML responsiveness. Streaming data is unbounded, unordered, and immutable, coming from various sources, and is essential for applications like fraud detection, inventory management, and customer personalization. Despite its importance, many organizations face challenges in adopting streaming architectures due to the complexity and expertise required. Managed services, like Confluent Cloud, offer solutions by simplifying the deployment and management of streaming platforms, ensuring scalability, reliability, and compliance. These services enable businesses to easily integrate streaming data into their operations, thereby enhancing agility, reducing costs, and delivering better customer experiences. As data streaming becomes the norm, companies that fail to adopt these technologies risk falling behind in the digital transformation race.

Australia’s Alex Bank, a digital challenger bank founded in 2018, uses event-based architecture and real-time data streaming to offer personal loans, savings accounts, and term deposits while aiming to make banking simpler, faster, fairer, and more human. After pandemic-related licensing delays, the bank replaced its technology stack, rebuilt customer origination systems, launched its personal loan product, and received its full banking license in December 2022. Its platform, built with Kafka and Confluent, combines data from broker and direct channels into standardized streams, enabling the bank to connect technical events to business transactions, maintain interoperability with legacy financial systems, and analyze fraud-related activity in real time. Alex Bank reports that its small engineering team built the platform in about six months, that it processes roughly 250 million monthly security events, and that it can collect thousands of customer attributes in under 40 seconds to support credit decisions. The bank also cites growth in lending, broker participation, and deposits, while noting that rapid automated decisions must be accompanied by customer education and transparency to build trust in a high-stakes financial process.

Apache Flink has gained significant popularity due to its unified stream and batch processing framework, which provides high resource efficiency, massive throughput with low latency, and robust state handling. The platform's flexibility allows it to be tailored to a wide range of workloads, including streaming and batch workloads. Apache Flink offers four different APIs, each tailored to different users and use cases, providing fine-grained control over data processing. The platform unifies stream and batch processing, allowing for consistent semantics across real-time and historical data processing use cases. Additionally, Flink has been hardened in production with various features such as a flexible metrics system, extensive observability, and savepoints. Its integration with Apache Kafka provides a seamless experience for stream processing, leveraging the latter's core streaming data storage layer. Many leading innovators have adopted both Kafka and Flink in their data stack, making it an attractive choice for organizations looking to build powerful data streaming architectures.

Niki Kapsi, a Commercial Account Executive at Confluent, describes her transition from Paris to Palo Alto and then Barcelona, where she moved from a Sales Development Representative role into managing commercial relationships across Greece, Cyprus, and the Balkans. She chose Confluent over larger companies because she saw significant business potential in Kafka and its ability to affect organizations across industries, despite initially lacking technical expertise. Kapsi views her role as entrepreneurial, acting as the leader of her territory by finding customers, researching accounts, personalizing outreach, coordinating events and partnerships, supporting co-selling, and maintaining customer relationships. She characterizes Confluent’s culture as one that promotes learning, creativity, professional growth, motivation, and respect across its U.S. and European teams, while humorously suggesting that the company open an office in Greece.

Camp Confluent was a three-week virtual learning program held July 10–28 that introduced participants to data-streaming concepts, Apache Kafka, Confluent Cloud, and real-time analytics through prerecorded sessions, demonstrations, customer examples, partner presentations, and a live Q&A. The first week focused on event-driven microservices and migration from monolithic applications, including BigCommerce’s Kafka scaling experience; the second covered building visual streaming pipelines for cloud databases and data warehouses; and the final week featured AWS and Imply sessions on actionable insights and real-time analytics applications. Weekly viewing challenges offered promotional rewards, while the closing session drew 145 participants from several countries and addressed topics such as connectors, Flink integration, generative AI, and Kafka use cases. The post also includes attendee and organizer testimonials and promotes Confluent Cloud and the Current 2023 Kafka Summit.

Confluent has become an official sponsor of Let’s Encrypt, which it has relied on since the beginning of its cloud platform for automated, open, and free TLS/SSL certificate issuance. As Confluent Cloud expanded to support varied networking models, including Confluent Cloud Networks and Private DNS Resolution, it replaced a monolithic, region-wide certificate approach with centralized certificate management designed to handle configurable endpoint and Subject Alternative Name requirements. The new system uses certificate schemes to generate domain lists based on cloud, region, and network details, then pre-provisions and pools certificates in a secret store to avoid DNS-challenge latency during resource provisioning and reduce exposure to temporary certificate-provider outages. Built on the ACME-compatible Lego library, the service abstracts DNS provider differences across cloud platforms, automates renewals, monitors certificates approaching expiration, synchronizes renewed certificates to clusters, and retains the option to use a backup ACME-compatible certificate authority. Confluent now obtains tens of thousands of certificates weekly through Let’s Encrypt APIs, enabling secure connectivity while allowing engineering teams to focus on additional networking capabilities.

Confluent has earned AWS’s Financial Services Competency, recognizing its cloud-based data streaming capabilities and customer success in helping banks, insurers, capital markets firms, and payment processors innovate through real-time data integration. Its platform connects legacy, on-premises, multicloud, and edge systems with AWS services to support fraud detection, cybersecurity and compliance, trading data platforms, and personalized customer experiences. Confluent highlights customers including 10x Banking and Judo Bank, which use Confluent and AWS to modernize systems, adopt event-driven architectures, and improve operational capabilities. For fraud prevention, the company emphasizes aggregating structured, unstructured, historical, and streaming data to create continuously updated risk scores, detect threats, and maintain auditable activity records. Confluent also cites its AWS Data & Analytics Software Competency, several AWS service-ready designations, and more than 120 prebuilt connectors for services including Amazon S3, Redshift, Kinesis, RDS, and DynamoDB.

Real-time AI applications are becoming increasingly essential across various industries, necessitating access to up-to-date data to provide accurate and responsive user experiences. Confluent and Rockset together form a powerful architecture for enabling real-time AI by combining Confluent's data streaming capabilities with Rockset's vector search functionality. This combination is crucial for applications like Whatnot's live auction platform, which relies on real-time data to recommend live streams effectively. Confluent Cloud provides a comprehensive data streaming solution that integrates seamlessly with various systems, while Rockset offers low-latency, high-concurrency query capabilities, making it ideal for real-time AI applications. Whatnot's use of this technology stack has significantly improved their recommendation engine, allowing for personalized suggestions in real-time and supporting their rapid growth. The synergy between Confluent and Rockset exemplifies how businesses can leverage real-time data to enhance AI-driven applications efficiently and at scale.

Apache Kafka is a distributed computing system that enables multiple servers to work together for speed, durability, and scalability. It uses a cluster structure, where brokers manage load balancing, replication, and stream decoupling, and partitions are used to balance the workload across the cluster. Data is replicated across multiple servers to ensure stability and availability. Kafka balances load through load balancing and partitioning, allowing it to handle large volumes of data and provide low latency. The system uses a leader-follower model for replication, where followers replicate writes from the leader node to maintain data consistency. In case of failure, KRaft, a metadata management system, is used to manage the cluster's metadata and trigger leader elections. Kafka can be run on-premise or in the cloud, providing developers with flexibility in choosing their deployment option. The system also uses events as a fundamental concept, enabling event-driven design, event sourcing, designing events, and event streaming.

Apna, Glance, and Meesho describe how data streaming with Apache Kafka and Confluent Cloud supports their rapid growth, real-time services, and customer experiences in India. Apna replaced monolithic systems with an event-driven microservices architecture that uses Kafka for communication, scheduling, pipelines, and services such as job matching and application tracking, while Confluent Cloud provides managed scalability and high availability. Glance processes large volumes of lock-screen interaction data to tailor content to user interests, using Kafka to decouple data producers and consumers while relying on managed tooling for schema governance, lineage, observability, and secure data access. Meesho uses Kafka as the first destination for ingested data supporting recommendations, seller and consumer experiences, and business decisions, with Confluent Cloud helping it accommodate demand spikes, elastically scale infrastructure, and reduce operational overhead. The examples present data streaming as a foundation for scalable architectures, data democratization, and faster development, while also noting governance and infrastructure-management challenges as streaming use expands.

The Confluent maturity curve outlines five levels of adoption for data streaming technologies like Apache Kafka. The first three levels (1-3) are characterized by bottom-up, tech-led adoption, while the fourth level (4) requires top-down, business-led adoption. Level 4 involves coordinating disparate teams, implementing a Center of Excellence, and managing data as a product to achieve economies of scale. To move from Level 3 to 4, organizations need education and awareness from the business, budget or solid business case outlining real benefits, senior-level sponsorship, and a change management approach. The Confluent maturity curve is being revised to emphasize the step from Level 3 to 4, with recommendations including writing a strategy or internal proposal, determining metrics and ROI, and implementing a people, process, and technology layer to support this shift. Organizations can benefit from managing data like a product, driving innovation, cost allocation, and risk management, while also reducing costs associated with managing data. Change management is critical for this transformation, involving identifying stakeholders, training, education sessions, tech talks, and new performance metrics.

Apache Kafka’s Java producer uses the Murmur2 hashing algorithm by default to assign keyed records deterministically to partitions, while many non-Java clients built on librdkafka commonly use CRC32-based partitioning, causing identical keys to land in different partitions. This difference can break co-partitioned operations such as Kafka Streams or ksqlDB joins, as illustrated by a Python-produced user topic and a ksqlDB-produced orders topic whose matching user IDs often fail to join because their records are distributed differently. To prevent these mismatches, librdkafka-based producers can be configured with the Java-compatible `murmur2_random` partitioner, which also matches Java’s handling of null keys, or existing topics can be repartitioned through ksqlDB into a new Murmur2-partitioned topic. The discussion notes that Apache Flink may avoid this specific issue because it typically repartitions data when ingesting it.

The text discusses the integration of AWS Lambda with Kafka topics to create scalable, event-driven architectures suitable for various industries, highlighting two primary integration patterns: using AWS Lambda as a sink connector and configuring Kafka as an event source. The Lambda Sink Connector allows synchronous or asynchronous invocation of functions, offering flexibility in workload management, while the event source mapping pattern automatically invokes Lambda functions based on message flow, supporting auto-scaling and load monitoring. The text provides detailed insights into each pattern's operational features, such as message processing, error handling with dead-letter queues, and parallel task executions. It also presents real-world use cases in healthcare and email marketing, illustrating the practical applications of these integrations. Additionally, the text mentions Confluent Cloud's support for multiple Lambda functions and cost-effective scaling, with an example from BT Group's use of Confluent for real-time data streaming in telecom.

Alex Lamy, Confluent’s Regional Sales Director for France, leads a sales organization serving Belgium, France, Luxembourg, Switzerland, and the Maghreb after joining the company five years earlier from MongoDB, where he recognized Kafka’s growing strategic importance. He initially helped establish Confluent’s go-to-market presence in France as a sales representative before becoming the leader of the French sales team in 2021. Lamy describes his role as focused on empowering team members, supporting customer success, and communicating Confluent’s value to organizations adopting Kafka, with team performance replacing individual sales targets as his primary measure of success. He characterizes Confluent’s culture as one based on trust, autonomy, collaboration, and mutual support, while noting that growth has required more structure. As a leader, he aims to create conditions, opportunities, and a healthy environment in which diverse employees can succeed, and he uses in-person off-site events, including activities near his home in Annecy, to strengthen connections within the remote-first organization.

Founded in 2016, Factorial expanded from basic HR software into an integrated HR, finance, and people-operations platform serving more than 8,000 businesses across 65 countries, but its original AWS-based batch data lakehouse could not provide the fresh data or low-latency queries needed for user-facing features. Scheduled MySQL snapshots, Spark processing, S3 storage, and Athena queries supported internal reporting but left data hours or days old and query responses ranging from seconds to minutes. To avoid separate developer-built systems and the operational burden of self-managed infrastructure, Factorial adopted Confluent Cloud and Tinybird, using MySQL change data capture through Debezium to stream changes into Kafka and Tinybird to enrich real-time streams with historical data and expose them through low-latency APIs. The managed architecture reduced data freshness to seconds and average production query times to under 50 milliseconds, while allowing a two-person data team to launch its first production feature within a month and more than 12 real-time user-facing features over the following six months.