January 2024 Summaries

Confluent announced the global semifinalists for its Data Streaming Startup Challenge, a competition for startups under five years old with less than $10 million in venture funding that build real-time applications using Confluent Cloud. Nearly 100 companies from 22 countries participated, applying data streaming to sectors including blockchain, computational chemistry, cannabis e-commerce, transportation, road safety, emissions monitoring, gaming, AI metering, e-commerce, mental health, and digital twins. The featured companies use Confluent and Kafka technologies for functions such as processing high-volume blockchain data, enabling event-driven services, aggregating usage and gaming events, monitoring operations in real time, and delivering personalized support. Each semifinalist receives $5,000 in Confluent Cloud credits and tickets to Kafka Summit London, while the eventual champion can receive a $500,000 Confluent investment and opportunities to pitch major venture capital firms; three finalists will present at Kafka Summit London on March 19.

Confluent's Partner Awards 2023 recognize partners who have demonstrated innovation and commitment in harnessing the power of real-time data, with a focus on collaborative efforts to drive transformative business solutions. Confluent itself has been named Microsoft's 2024 OSS on Azure Global Partner of the Year for its commitment to delivering outstanding open source-based applications and infrastructure solutions on Microsoft Azure. The award reflects the growing importance of data streaming as a critical foundation of modern business, with technology now serving as the business itself. By collaborating with partners, Confluent aims to provide joint customer value while driving transformative business solutions through real-time data.

Confluent is leveraging its event-driven microservices within the platform to alert internal teams with timely and actionable insights. The company identifies critical events such as creating a Confluent Cloud cluster, starting a new account or cloud organization, releasing new versions of products, aggregated usage patterns, customer health snapshots, and more transactional events like new opportunities or approaching the end of existing agreements. These events are captured from various systems using Confluent's managed connectors or Cluster Linking, which replicates data from one cluster to another. The events are then enriched with contextual data using Kafka Streams, a stream processing tool that allows for real-time filtering and formatting alerts to be sent via customer Slack applications. The business challenge lies in taking these insights and using them to accelerate business objectives with some action, such as recommending resources or activities to enable customers' success. Confluent uses its data streaming platform to decouple components and leverage managed connectors, stream processing tools, and consumer microservices to deliver timely actions and automate processes.

Data streaming platforms are becoming increasingly important in today's fast-moving world, with companies like Walmart and 10x Banking using them to manage their data effectively. Confluent, a leader in the field, has been recognized by Forrester as a leader in both The Forrester Wave™: Cloud Data Pipelines, Q4 2023 and The Forrester Wave™: Streaming Data Platforms, Q4 2023. Confluent's data streaming platform is built on Apache Kafka® and offers a complete solution for data streaming, including stream processing with Apache Flink®. The company has also expanded its offerings to include managed connectors and governance tools, making it easier for organizations to build efficient cloud data pipelines that deliver real-time insights. Confluent's commitment to innovation and customer success is evident in their recent acquisitions and leadership changes, as they continue to drive the data streaming revolution forward.

Telemedicine has significantly expanded global access to healthcare by offering services like video consultations and remote monitoring, but its efficiency is heavily reliant on robust data infrastructures. The text discusses the challenges of implementing effective telemedicine services, such as maintaining security and compliance with regulations like HIPAA and GDPR, managing scalable data pipelines, and integrating diverse healthcare data systems. Data streaming, particularly with platforms like Confluent Cloud based on Apache Kafka, is presented as a solution to these challenges, offering a cloud-native, scalable, and reliable infrastructure that can handle high volumes of complex data. The platform supports seamless data integration and processing, enhancing telemedicine services by enabling real-time communication and interoperability through the use of standards like HL7 FHIR. Additionally, Confluent Cloud's features, including pre-built connectors and stream processing capabilities, facilitate the creation of efficient telemedicine applications, ensuring data security and compliance while supporting the growing demands of remote healthcare services.

The new BigQuery Sink V2 connector, a significant upgrade to the existing BigQuery Sink connector, leverages Google's recommended Storage Write API and adds OAuth 2.0 support, enabling customers to stream records in parallel through an internal thread pool and supporting multiple data formats. This updated connector aligns with Google Cloud's best practices and recommendations, ensuring customers benefit from the latest technologies. With improved efficiency, cost-effectiveness, and flexibility, the BigQuery Sink V2 connector simplifies the data ingestion process for users, while also reducing the total cost of ownership. The new connector offers enhanced security features, scalability, and support for multiple data formats, including Avro, JSON Schema, Protobuf, and JSON schemaless input data formats. Additionally, customers can migrate their existing pipelines to use the new connector by creating a test environment and running it against a test pipeline before moving their existing pipelines. The new connector is available on Google Cloud, Confluent Cloud on AWS, and Azure, with a demo webinar and documentation available for users to learn more about its capabilities.

Confluent’s command-line interface helps developers and administrators manage Apache Kafka clusters, Apache Flink compute pools, and related resources efficiently through composable commands that support automation, scripting, and detailed configuration. To simplify recurring multi-step workflows that would otherwise require external scripts, Confluent introduced a plugin framework that lets users create discoverable custom CLI commands from standalone executables written in Go, Python, or Bash and named with the `confluent-` prefix. Plugins can be installed from the user’s PATH or searched and installed through Confluent’s public plugin repository. One example, the `confluent-flink-quickstart` plugin, automates the setup required to begin using Confluent Cloud for Apache Flink, including provisioning compute resources, configuring Schema Registry and Kafka, obtaining credentials, selecting databases, and optionally generating sample data, reducing a sequence of dependent commands to a single workflow.

Confluent’s 2024 resolution is to simplify Apache Kafka development and help programmers build event-streaming applications that produce actionable insights and real-time experiences. The post curates introductory tutorials for Kafka development in Java, Python, Go, .NET, Node.js, C/C++, REST Proxy, and Spring Boot, alongside more advanced materials such as a .NET-focused course and a broader overview of Kafka tools, languages, and examples for reading, processing, and writing cluster data. It also recommends articles on Kafka-based applications in Python, C#/.NET, C++, Rust, and Node.js, and promotes Kafka Summit 2024 events in London on March 19–20 and Bangalore on May 2 for members of the Kafka community.

Confluent announces an integration with Pinecone serverless, a vector database architecture designed to reduce infrastructure management and support low-latency vector search across large data volumes through separated read, write, and storage layers, blob-storage-based clustering, and multi-tenant compute. The partnership is positioned as a way to help organizations build retrieval-augmented and other GenAI applications using continuously updated enterprise data, addressing concerns that batch-based data pipelines can produce stale, inconsistent, and difficult-to-govern information. Confluent describes its cloud-native streaming platform as providing real-time data ingestion, processing, governance, and deployment flexibility across cloud, on-premises, and hybrid environments, with Apache Flink used to enrich and transform streams before vector database updates. Its preview Pinecone Sink Connector transfers data from Confluent Cloud topics, creates vector embeddings using supported Azure OpenAI models, and writes them to Pinecone indexes with at-least-once delivery; it supports several structured and schemaless data formats but has regional, platform, and format limitations. Both companies promote free trials, with Confluent offering new Cloud users credits for their first 30 days.

Order management systems (OMS) have transitioned from manual record-keeping to sophisticated web-based platforms that automate tasks, reduce errors, and enhance inventory management across various sales channels. These systems are crucial for efficient order fulfillment and customer experience but face challenges such as data inconsistencies, fraud detection, and customer disconnects. Confluent, a real-time data streaming platform based on Apache Kafka, addresses these challenges by providing seamless data integration and transformation, enabling businesses to maintain accurate data, detect fraud in real-time, and improve customer experiences with personalized recommendations. Through its robust capabilities, Confluent helps businesses optimize their order management processes, ensuring order integrity, real-time inventory management, and enhanced collaborative efficiency across organizational teams, ultimately leading to cost savings and a competitive edge in the market.

Confluent Cloud’s OAuth support, released in December 2022, enables organizations to integrate third-party identity providers such as Azure Active Directory Domain Services to centrally authenticate and authorize applications accessing Kafka clusters. Applications use the OAuth client credentials grant to obtain a signed JWT from the identity provider, present it through the SASL OAUTHBEARER mechanism, and are authorized by Confluent Cloud through identity pools and RBAC role bindings based on token claims. Configuration requires registering an identity provider with its issuer and JWKS endpoints, defining identity pools with claim-based filters, and configuring supported Kafka clients with the token endpoint, client credentials, target cluster, and identity pool ID. The Azure AD example details registering a Confluent Cloud application, creating app roles, enabling version 2 access tokens, registering client applications and secrets, granting permissions, and mapping JWT roles to Confluent Cloud identity pools. Identity pools can reduce reliance on individual service accounts and API keys, support shared authorization for common resources, or enforce application-specific access, including distinct permissions across development and production environments.

The Department of Defense aims to become a data-centric organization by treating data as a strategic asset that must remain interoperable and accessible across enterprise cloud, tactical edge, and disrupted, disconnected, intermittent, and low-bandwidth environments. A data fabric, defined as an integrated layer that uses metadata and continuous analytics to connect reusable data across environments, is presented as a way to replace fragmented point-to-point pipelines with scalable architecture that supports edge computing and AI-enabled operations. The Navy is applying this approach through its Logistics Information Technology program, which integrates supply and logistics data across key maintenance systems, and its Logistics Information Naval Connector platform, which provides standardized hosting for logistics applications. Confluent describes its role as a data streaming platform for these programs, consolidating siloed information on maintenance, supplies, and assignments to improve maintenance scheduling, mission planning, and predictive logistics while reducing system-integration time and costs.

Confluent Cloud has significantly enhanced its performance, achieving speeds up to 10 times faster than Apache Kafka through the use of Kora, a cloud-native Kafka engine. Despite potential challenges in comparing performance due to Confluent Cloud's additional services like observability and auditing, Kora has been optimized to maintain low and consistent latency even under varying workload conditions and infrastructure disruptions. The blog post details how Confluent Cloud outperforms Apache Kafka across various workload profiles, achieving up to 16x improvements in p99 end-to-end latencies. The architecture has evolved with features like increased batching, parallelism, and a new replication protocol to ensure reliability and performance stability. Continuous optimizations have been implemented, such as improving the self-balancing algorithm and handling infrastructure degradation to minimize latency spikes. The post highlights real-world examples and customer feedback, illustrating the impact of these enhancements, and invites users to experience the improved performance of Confluent Cloud with the Kora engine.

BT, a multinational telecommunications company, built the Vena orchestration system to manage their broadcasting infrastructure. The existing network was straining and needed to be replaced quickly within a tight budget. BT leveraged advances in Software Defined Wide Area Network (SD-WAN) technology to automate site provisioning, service ordering, and service assurance monitoring and reporting. They chose to build the software system from scratch using open source components, allowing for full ownership of the software stack and flexibility to accommodate unanticipated features. The Vena platform carries services for all major broadcasters in the UK and is built on top of Apache Kafka, providing real-time data processing, scalability, event-driven architectures, efficiency, availability, and reliability. Confluent Platform clusters deployed with Terraform enable automated deployment, near-real-time geo-replication, and high resilience. The system also uses Confluent Control Center for detailed monitoring and security features such as encryption and fine-grained access control. Vena has successfully implemented two vital use cases that demonstrate its ability to handle various types of data and integrate disparate sources in real-time.