January 2026 Summaries

Data streaming has emerged as a crucial technology for enabling event-driven microservices, allowing companies to efficiently handle large-scale data processing through composable applications. Apache Kafka plays a central role in this landscape by providing an asynchronous publish/subscribe architecture that facilitates data sharing across enterprises. While traditional ETL data integration offers simplicity and familiarity, it often results in isolated, siloed data, lacking the flexibility and reusability that streaming data integration offers. Streaming data integration, particularly with Kafka, supports real-time and batch processing, allowing data to be enriched and shared across multiple destinations. This model provides a more interconnected and flexible approach to data management, eliminating the need for numerous point-to-point ETL connections and enabling seamless integration across various platforms and use cases. By keeping data in motion, streaming data integration enhances the ability to perform real-time analytics and supports a unified view of both real-time and historical data, offering a powerful alternative to the traditional ETL paradigm.

Securing API interactions in Confluent Cloud is crucial, with the platform offering cloud API keys and resource-specific API keys to manage authentication across its services. Cloud API keys provide organization-wide access for administrative tasks, while resource-specific API keys offer granular permissions for specific resources like Kafka clusters, ksqlDB, and Schema Registry, enabling fine-tuned security. Confluent Cloud distinguishes between user accounts, suitable for personal administrative tasks, and service accounts, which are ideal for applications and automated processes due to their persistent nature and separation from individual users. Additionally, External OAuth provides an alternative to API keys, granting access through short-lived tokens while leveraging existing identity management systems for enhanced security and simplified access management. By understanding and utilizing these security mechanisms, users can effectively manage and secure data streaming operations within Confluent Cloud.

Confluent emphasizes the importance of trust in customer relationships by providing a secure and transparent platform through its Confluent Trust Center. This dedicated portal offers on-demand access to security, privacy, and compliance documentation, facilitating due diligence for prospects and simplifying continuous compliance for existing customers. The Trust Center accelerates procurement processes with instant access to compliance documents and standardized security assessments, while also aiding customers in maintaining audit readiness and adapting to global regulatory changes. The introduction of "Ask AI" enhances user experience by providing quick, accurate responses to security queries, reinforcing Confluent's commitment to transparency and empowering customers to navigate their security and compliance needs effectively.

A global investment bank collaborated with Confluent to achieve ultra-low latency in their trading pipelines, reaching a sub-5ms, 99th percentile latency at a rate of 1.6 million messages per second, crucial for real-time trading in global capital markets. This was accomplished through a meticulous approach involving architectural discipline, comprehensive monitoring, and strategic configurations in a multi-data center deployment, focusing on every aspect of the Kafka message path to mitigate latency outliers. The project emphasized the significance of understanding single-partition latency baselines, addressing infrastructure bottlenecks, and employing a scientific, iterative tuning process. Utilizing tools like the OpenMessaging Benchmark, the team was able to systematically enhance system performance, ensuring robust order guarantees and high throughput, which are vital for mission-critical financial applications. This case study provides valuable insights into the challenges and best practices for achieving low-latency streaming with Kafka at scale, highlighting the importance of infrastructure upgrades, such as enterprise SSDs and ZGC, and the role of reproducible benchmarking in performance optimization.

Thunai has developed an agentic AI platform designed to enhance customer support by automating routine tasks and improving the efficiency of support teams. The platform integrates real-time data streaming, utilizing Confluent's capabilities to maintain a continuously updated knowledge base called Thunai Brain. This ensures AI agents have access to fresh, reliable context, enabling them to automate Level 1 support tasks and reduce the workload on human agents. By leveraging live data from various channels, including voice, chat, and email, Thunai can provide consistent, timely customer interactions while minimizing operational inefficiencies. The adoption of a streaming-first architecture has allowed Thunai to achieve significant deflection rates for customer queries, decrease resolution times, and enhance overall customer experiences. This approach sets a foundation for future expansion into advanced AI capabilities without the need for rearchitecting.

Confluent, a remote-first company, emphasizes the importance of connection and purpose in the workplace by offering flexibility in working models and prioritizing in-person onboarding sessions for new hires. This approach helps new employees understand the company's mission and fosters a sense of belonging from the outset, bridging the gap between remote work and real connection. Confluent's commitment to employee growth extends beyond onboarding through its career development program, Growth in Motion, which provides continuous feedback, clear career paths, and skill development opportunities. By investing in these initiatives, Confluent aims to create a work environment where technology leads, high-impact work is standard, and employees find genuine purpose in their roles.

Streaming data with Apache Kafka involves real-time data movement, but the transition from Apache ZooKeeper to KRaft as the control plane poses significant challenges, especially with ZooKeeper's end of support in Kafka 4.0. Confluent Cloud offers a streamlined solution for this migration, providing a cloud-native data streaming platform that reduces Kafka costs by 40%–70% and eliminates operational burdens with a 99.99% uptime SLA. The platform's Kafka Copy Paste (KCP) tool simplifies migration planning and execution, while Cluster Linking ensures secure and efficient data transfer between Kafka clusters. Beyond one-time migrations, Confluent Cloud facilitates ongoing data unification across global and hybrid environments, enhancing data availability and regulatory compliance. The platform offers flexible deployment across multiple cloud providers, integrated security features, and a variety of connectors, making it a comprehensive solution for modern data streaming needs.

Current, the successor to Kafka Summit, is an essential event for data streaming professionals, marking its evolution alongside the growing community since its inception in 2017. Originally focused on Apache Kafka, the event has expanded to encompass the broader ecosystem that has developed around Kafka, which now powers applications, startups, and careers in data streaming. As the technology landscape is further transformed by AI, Current adapts by offering a wider range of discussions and practices, including a Data Streaming World Tour (DSWT) with stops in India and an expanded main event in San Francisco, alongside other focused gatherings like a mini-Current dedicated to data engineering. These changes aim to foster a more inclusive and collaborative environment for developers, architects, vendors, and executives to continue exploring the potential of data streaming and AI, while maintaining its position as a premier global event in the field.

Apache Kafka Connect serves as a crucial tool for integrating Kafka with various data systems, allowing for seamless data movement between sources and sinks in distributed environments. It provides a robust framework for building scalable and fault-tolerant connectors that handle real-time data streaming. While many pre-built connectors are available through platforms like Confluent Hub, there are instances where custom connectors are necessary to address unique business requirements, unsupported data sources, or complex data processing needs. Building a custom connector involves understanding Kafka Connect's architecture, including connectors, tasks, and workers, as well as mastering necessary skills such as Java programming and REST APIs. The development process includes setting up a development environment, implementing the connector's core logic, packaging, deploying, and testing it within a Kafka Connect cluster. Challenges like maintaining data schema consistency, ensuring fault tolerance, and integrating security features must be addressed to achieve a reliable and efficient connector. By leveraging Confluent's tools and best practices, developers can optimize performance, ensure robust error handling, and maintain compatibility across different environments, ultimately enhancing data integration capabilities in real-time streaming architectures.