July 2024 Summaries

The new release of Confluent Platform 7.7 enhances security while reducing operational burden by managing application identities and credentials through your own OIDC identity provider with OAuth, allowing for streamline authentication with one source of identity and maintaining compliance without sacrificing efficiency. This update also includes support for Apache Flink, a stream processing framework that has emerged as the de facto standard, but can be challenging to manage due to its operational complexity. Confluent Platform for Apache Flink provides long-term support for on-premises or private-cloud Flink workloads, consolidating risk and providing expert guidance from industry experts. Additionally, the release introduces a new HTTP Source connector that guarantees records are delivered at least once to Kafka topics and supports multiple tasks, as well as support for migrating from ZooKeeper to KRaft using Confluent for Kubernetes.

Modern data platforms are crucial for businesses to extract valuable insights from various sources of data. Data streaming and batch processing frameworks help provide these insights, with stream processing enabling real-time reactions to changing events. Managed Apache Kafka clusters on Confluent Cloud have become the standard for building sophisticated data streaming applications. Flink AI (FLIP-437) opens up a new paradigm for Flink SQL on Confluent Cloud by making machine learning models first-class citizens, allowing integration between AI workloads and streaming data in real time. This feature is currently in early access mode but will be made generally available soon. The use of generative AI (GenAI) large language model (LLM) endpoints and narrow AI models can enrich the quality of streaming data instantly, enabling smarter applications through real-time ML predictions.

Apache Kafka 3.8.0 has been released, featuring several new features and improvements, including support for compression level configuration, deprecation of Log4J Appender, and enhancements to the Consumer Rebalance Protocol. The release also includes various bug fixes, new metrics, and improved state store iterator metrics. Additionally, Apache Kafka 3.8.0 introduces a mechanism to restrict files accessed by File and Directory ConfigProviders, allows producer and consumer clients to rebootstrap, and adds support for Docker images for GraalVM-based Native Kafka Broker. The release also includes several KIPs (Kafka Improvement Proposals) that introduce new features, such as Shareable State Stores, customizable task assignment for Streams, and improved state store iterator metrics.

Confluent has introduced several enhancements to its platform, including interactive tables in Flink SQL Workspaces that allow users to quickly explore and visualize query results, improved stream exploration and profiling, and message filtering and search capabilities for troubleshooting. Private networking support for Flink provides secure data streaming workloads, while serverless Enterprise clusters offer elastic autoscaling and cost-efficiency. The Connect with Confluent technology partner program has expanded to include seven new partners, including Amazon EventBridge, Couchbase, and Neo4j, offering native integrations and simplified data streaming use cases. Additionally, Confluent has introduced custom offset management for migrating from self-managed to fully managed connectors, as well as a "Bring Your Own Key" feature that allows customers to manage their own encryption keys. The company also offers a migration accelerator program to help customers transition to Confluent Cloud, and has expanded its library of partner solutions and GitHub repositories.

City of Hope uses a combination of technologies including Apache Kafka, Confluent Cloud on Microsoft Azure, Flink, MongoDB, Epic electronic health record system, and Confluent's rich ecosystem of connectors to build a real-time infrastructure for its sepsis prediction model. The model uses data streaming to predict the risk of sepsis in bone marrow transplant patients, reducing sepsis and ICU escalation rates. City of Hope's goal is to help clinicians make data-driven clinical decisions using real-time data, and the system has been successful in doing so. Confluent Cloud on Microsoft Azure enables the organization to better manage its real-time infrastructure, providing clarity into the system and enabling quick problem-solving. The use of Kafka serves as a backup solution, ensuring uninterrupted processing. By embracing a research-first approach and nurturing relationships with internal stakeholders, City of Hope has been able to build a scalable and reliable system that supports future application development.

As a Customer Marketing Manager at Confluent, Alexri Patel-Sigmon is driven by her passion for customer storytelling and the impact of innovation in real-time data streaming technology. With two years of experience at the company, she has grown both personally and professionally, developing technical proficiency, adaptability, leadership, and innovation skills. Confluent's collaborative culture fosters a sense of partnership and trust with customers, leading to long-term collaboration and mutual success. Alexri envisions significant growth in her role over the next few years, advancing to a more senior position and contributing to strategic initiatives that drive the company's growth and success. The unique aspect of Confluent's culture is its people, who bring together diverse perspectives to achieve common goals.

This summary highlights a discussion on building a GenAI model with Confluent, Flink, and MongoDB, which uses Apache Kafka as its underlying messaging system. The project aims to settle the debate on tabs vs spaces by creating an app that produces and consumes vote counts from a Kafka topic, utilizing the confluent-kafka-javascript client for seamless integration with Confluent's cloud offering. The code demonstrates how to use the promisified API for consuming events, updating state in the backend, and displaying the results in a user-friendly interface using vanilla JavaScript and CSS. The project showcases opportunities for growth, such as implementing visualizations or fingerprinting users, and provides resources for learning more about Kafka and Confluent-kafka-javascript.