January 2024 Summaries

Confluent's Partner Awards 2023 recognize partners who have demonstrated innovation and commitment in harnessing the power of real-time data, with a focus on collaborative efforts to drive transformative business solutions. Confluent itself has been named Microsoft's 2024 OSS on Azure Global Partner of the Year for its commitment to delivering outstanding open source-based applications and infrastructure solutions on Microsoft Azure. The award reflects the growing importance of data streaming as a critical foundation of modern business, with technology now serving as the business itself. By collaborating with partners, Confluent aims to provide joint customer value while driving transformative business solutions through real-time data.

Confluent is leveraging its event-driven microservices within the platform to alert internal teams with timely and actionable insights. The company identifies critical events such as creating a Confluent Cloud cluster, starting a new account or cloud organization, releasing new versions of products, aggregated usage patterns, customer health snapshots, and more transactional events like new opportunities or approaching the end of existing agreements. These events are captured from various systems using Confluent's managed connectors or Cluster Linking, which replicates data from one cluster to another. The events are then enriched with contextual data using Kafka Streams, a stream processing tool that allows for real-time filtering and formatting alerts to be sent via customer Slack applications. The business challenge lies in taking these insights and using them to accelerate business objectives with some action, such as recommending resources or activities to enable customers' success. Confluent uses its data streaming platform to decouple components and leverage managed connectors, stream processing tools, and consumer microservices to deliver timely actions and automate processes.

Data streaming platforms are becoming increasingly important in today's fast-moving world, with companies like Walmart and 10x Banking using them to manage their data effectively. Confluent, a leader in the field, has been recognized by Forrester as a leader in both The Forrester Wave™: Cloud Data Pipelines, Q4 2023 and The Forrester Wave™: Streaming Data Platforms, Q4 2023. Confluent's data streaming platform is built on Apache Kafka® and offers a complete solution for data streaming, including stream processing with Apache Flink®. The company has also expanded its offerings to include managed connectors and governance tools, making it easier for organizations to build efficient cloud data pipelines that deliver real-time insights. Confluent's commitment to innovation and customer success is evident in their recent acquisitions and leadership changes, as they continue to drive the data streaming revolution forward.

Telemedicine has significantly expanded global access to healthcare by offering services like video consultations and remote monitoring, but its efficiency is heavily reliant on robust data infrastructures. The text discusses the challenges of implementing effective telemedicine services, such as maintaining security and compliance with regulations like HIPAA and GDPR, managing scalable data pipelines, and integrating diverse healthcare data systems. Data streaming, particularly with platforms like Confluent Cloud based on Apache Kafka, is presented as a solution to these challenges, offering a cloud-native, scalable, and reliable infrastructure that can handle high volumes of complex data. The platform supports seamless data integration and processing, enhancing telemedicine services by enabling real-time communication and interoperability through the use of standards like HL7 FHIR. Additionally, Confluent Cloud's features, including pre-built connectors and stream processing capabilities, facilitate the creation of efficient telemedicine applications, ensuring data security and compliance while supporting the growing demands of remote healthcare services.

The new BigQuery Sink V2 connector, a significant upgrade to the existing BigQuery Sink connector, leverages Google's recommended Storage Write API and adds OAuth 2.0 support, enabling customers to stream records in parallel through an internal thread pool and supporting multiple data formats. This updated connector aligns with Google Cloud's best practices and recommendations, ensuring customers benefit from the latest technologies. With improved efficiency, cost-effectiveness, and flexibility, the BigQuery Sink V2 connector simplifies the data ingestion process for users, while also reducing the total cost of ownership. The new connector offers enhanced security features, scalability, and support for multiple data formats, including Avro, JSON Schema, Protobuf, and JSON schemaless input data formats. Additionally, customers can migrate their existing pipelines to use the new connector by creating a test environment and running it against a test pipeline before moving their existing pipelines. The new connector is available on Google Cloud, Confluent Cloud on AWS, and Azure, with a demo webinar and documentation available for users to learn more about its capabilities.

Order management systems (OMS) have transitioned from manual record-keeping to sophisticated web-based platforms that automate tasks, reduce errors, and enhance inventory management across various sales channels. These systems are crucial for efficient order fulfillment and customer experience but face challenges such as data inconsistencies, fraud detection, and customer disconnects. Confluent, a real-time data streaming platform based on Apache Kafka, addresses these challenges by providing seamless data integration and transformation, enabling businesses to maintain accurate data, detect fraud in real-time, and improve customer experiences with personalized recommendations. Through its robust capabilities, Confluent helps businesses optimize their order management processes, ensuring order integrity, real-time inventory management, and enhanced collaborative efficiency across organizational teams, ultimately leading to cost savings and a competitive edge in the market.

Confluent Cloud has significantly enhanced its performance, achieving speeds up to 10 times faster than Apache Kafka through the use of Kora, a cloud-native Kafka engine. Despite potential challenges in comparing performance due to Confluent Cloud's additional services like observability and auditing, Kora has been optimized to maintain low and consistent latency even under varying workload conditions and infrastructure disruptions. The blog post details how Confluent Cloud outperforms Apache Kafka across various workload profiles, achieving up to 16x improvements in p99 end-to-end latencies. The architecture has evolved with features like increased batching, parallelism, and a new replication protocol to ensure reliability and performance stability. Continuous optimizations have been implemented, such as improving the self-balancing algorithm and handling infrastructure degradation to minimize latency spikes. The post highlights real-world examples and customer feedback, illustrating the impact of these enhancements, and invites users to experience the improved performance of Confluent Cloud with the Kora engine.

BT, a multinational telecommunications company, built the Vena orchestration system to manage their broadcasting infrastructure. The existing network was straining and needed to be replaced quickly within a tight budget. BT leveraged advances in Software Defined Wide Area Network (SD-WAN) technology to automate site provisioning, service ordering, and service assurance monitoring and reporting. They chose to build the software system from scratch using open source components, allowing for full ownership of the software stack and flexibility to accommodate unanticipated features. The Vena platform carries services for all major broadcasters in the UK and is built on top of Apache Kafka, providing real-time data processing, scalability, event-driven architectures, efficiency, availability, and reliability. Confluent Platform clusters deployed with Terraform enable automated deployment, near-real-time geo-replication, and high resilience. The system also uses Confluent Control Center for detailed monitoring and security features such as encryption and fine-grained access control. Vena has successfully implemented two vital use cases that demonstrate its ability to handle various types of data and integrate disparate sources in real-time.