April 2024 Summaries

The creation of a single patient view (SPV) in healthcare requires addressing technical challenges such as integrating complex and sensitive data across disparate systems, ensuring interoperability, and maintaining regulatory compliance. To overcome these challenges, many organizations are adopting data streaming platforms like Apache Kafka and Confluent to securely process and share medical information in real-time. These platforms enable healthcare providers to build an SPV that powers a range of critical use cases, such as clinical trial matching services, while also improving patient outcomes by providing timely and accurate access to actionable patient data. With the benefits of fully managed and serverless data streaming platforms like Confluent Cloud, organizations can simplify their architecture, reduce maintenance time, and increase flexibility in delivering high-quality healthcare information.

The smart city revolution is transforming urban living by leveraging technology to address challenges such as congestion, unreliable public transport, and failing waste management systems. Data infrastructure plays a critical role in this transformation, with real-time data streaming enabling the creation of intelligent transportation systems, smart traffic light systems, and route optimization platforms. Confluent, a cloud-native data streaming platform, is being used by organizations to deliver various smart city applications, including demand-responsive transportation and smart street lighting. By processing large volumes of heterogeneous data in real time, Confluent enables smart city applications to make urban travel faster, cheaper, and safer while reducing its environmental impact.

Confluent’s kafka-docker-composer is a Python and Jinja2-based tool that generates Docker Compose configurations for local Confluent Platform and Apache Kafka testing, reducing the manual errors involved in assigning unique hostnames, ports, listeners, dependencies, and environment variables across multi-node clusters. It supports ZooKeeper- and KRaft-based deployments, configurable numbers of brokers, controllers, Schema Registry, Kafka Connect, ksqlDB, Control Center, Prometheus, Grafana, custom connector plugins, and rack-aware layouts for resilience testing. The tool is intended for scenarios ranging from simple single-broker setups to multi-data-center failover simulations, demonstrating behaviors such as broker or rack loss, replica distribution, and ZooKeeper limitations in two-data-center configurations. It also includes experimental shared KRaft mode, configurable cluster UUIDs, optional traffic-control-enabled images for injected network latency, and exposed monitoring interfaces, while future work may add security features and support for official open-source Kafka images.

Industry 4.0 is accelerating the adoption of smart factories through the integration of IoT, cloud computing, analytics, AI, and real-time data streaming. Siemens and automotive supplier Brose initially used open-source Apache Kafka to improve the movement of data from factory equipment to business systems, then adopted Confluent offerings to simplify integration, management, and scaling across global operations. Siemens moved to Confluent Cloud after legacy sales-system processes caused order delays and batch-based updates that took weeks, finding cloud deployment more cost-effective while improving product master-data update times through tools such as Kafka Streams and ksqlDB. Brose’s Future Pro Program established a data-driven strategy built around an on-premise streaming platform, though its diverse long-lived machine fleet, decentralized architecture, zero-trust security requirements, and coordination across 45 factories created operational complexity. Both organizations view secure data sharing, data quality controls, continuous monitoring, and potential data monetization as important next steps, particularly as manufacturers and customers increasingly seek access to industrial data.

This inaugural event brings together everything that makes Kafka Summit, well, Kafka Summit, including technical sessions, networking opportunities, the Expo Hall with sponsors and cool tricks, a closing reception, and social media fun. The author recommends selecting at least three sessions to attend, using the conference app to build a personal agenda, and exploring the Expo Hall in between sessions. Networking is encouraged, as well as attending the closing reception and participating in social media conversations using #KafkaSummit. The event will also be livestreamed for those who cannot attend in person, with recorded sessions available afterwards. Confluent is named Microsoft's 2024 OSS on Azure Global Partner of the Year, reflecting its commitment to delivering open source-based applications and infrastructure solutions on Microsoft Azure.

In the ever-evolving landscape of digital transformation, businesses seek robust platforms for handling data in motion, leveraging Apache Kafka as a de-facto standard, and Confluent's Accelerate with Confluent initiative propels seamless data flow and fosters collaborations between system integrators, partners, and clients. The initiative emphasizes collaborative efforts, aligning professional services capabilities to deliver comprehensive solutions that meet customer needs. Co-Delivery integrates expertise from both Confluent and certified partners, ensuring holistic solutions. Strategic Staffing enhances customer experience by incorporating subject matter experts, expanding delivery teams, and accelerating project timelines. Accelerate with Confluent offers multiple paths for partner engagement, including Subcontract, Co-Delivery, and Partner-Led, providing flexibility and allowing partners to choose the level of engagement that suits their preferences and capabilities. By focusing on customer needs, fostering innovation, and providing a comprehensive solution, Confluent empowers businesses to pioneer new frontiers in data management, unlocking unparalleled success in today's data-driven era.

Alejandro Martijena, Confluent’s Regional Sales Director for Latin America, describes how helping customers apply data streaming solutions to real-world and mission-critical challenges motivates his work. Since joining the company in 2017 as an inbound sales development representative, he has advanced through commercial and account executive roles to lead both enterprise and digital-native account teams across the region, developing leadership, customer engagement, and territory-growth capabilities. He credits Confluent’s culture of continuous learning, mentorship, autonomy, collaboration, and accountability with supporting his development, while emphasizing that customer interactions provide essential insight for tailoring solutions and building trust. Martijena also highlights the company’s top-down commitment to diversity, equity, and inclusion through diverse leadership, training, resources, and employee-led groups, and encourages prospective employees to ask how the organization fosters entrepreneurial ownership.

Confluent Schema Registry is an independent service that centrally stores and manages schema metadata—such as subjects, versions, IDs, and compatibility settings—for data exchanged through Apache Kafka, while Kafka itself stores the serialized messages. Producers communicate with Schema Registry over HTTP to register or retrieve schemas and IDs before serializing records and sending them to Kafka over TCP, while consumers retrieve messages from Kafka, read the embedded schema ID, fetch the corresponding schema from Schema Registry, and deserialize the payload. Confluent serializers for Avro, JSON Schema, and Protobuf prefix messages with a magic byte and four-byte schema ID, avoiding the overhead of including full schemas in every record; client-side caching limits repeated registry requests. The discussion uses Python Avro examples to illustrate schema registration, serialization, deserialization, subject naming, and optional use of the latest schema version, while recommending CI/CD-based schema management rather than producer-side automatic registration in production. It also describes schema-level data quality rules that can validate, transform, and govern data at ingestion, and demonstrates that Schema Registry can serialize and deserialize data without Kafka, although its primary value is supporting compatible, resilient, and governed Kafka-based streaming systems.

Confluent Cloud offers a portfolio of 80+ fully managed connectors that enable quick and reliable integration with popular data sources and sinks, connecting entire systems in real-time. To securely connect to these systems, Confluent provides various features such as public egress IPs, Service/Gateway endpoints, DNS Forwarding, PrivateLink, and Egress Access Points, which can be leveraged depending on the location of the data source/sink and the desired level of security. These features enable secure connections over public and private networks, reducing the risk of data breaches and other security threats. By utilizing Confluent Cloud's fully managed connectors with these secure networking options, users can ensure a reliable and secure integration with their existing data systems.

This week in Las Vegas, Google is hosting its annual Google Cloud Next ʼ24 conference, bringing together industry leaders to learn about and explore the newest innovations in and around the Google Cloud ecosystem. Confluent is sponsoring and attending Next and highlighting the capabilities of combining Google's world-class data, analytics, and GenAI technology stack with Confluent's industry-leading data streaming platform. The company has been named a 2024 Google Cloud Partner of the Year award winner for its dedication to revolutionizing data connectivity and enabling business innovation through seamless access to real-time data. Confluent's collaboration with Google Cloud represents a shared vision to drive innovation and deliver tangible results for customers, with a focus on driving innovation around customers' usage of real-time data. The partnership enables organizations to effortlessly stream data into leading services such as BigQuery, Cloud Storage, Cloud SQL, Dataflow, and more, creating cutting-edge, real-time AI applications. Confluent has also achieved the Cloud Ready designation for Google Distributed Cloud deployments, ensuring compatibility with edge or air-gapped use cases, and launched a fully managed BigQuery Sink V2 connector offering improved efficiency and cost-effectiveness. The company is also a launch partner for the Confluent Migration Accelerator and the Data Streaming for AI initiative, which leverages Google's extensive AI partnerships to expedite the creation of real-time AI applications.

Bergur Ziska, a Senior Customer Success Technical Architect at Confluent, transitioned from helping a financial institution adopt Confluent’s data streaming platform to advising customers directly on how to achieve technical and business value from streaming technologies. His work includes sharing best practices, conducting technical reviews, and developing architecture and migration strategies, while customer engagements provide insights into varied industry constraints and opportunities to improve products. Since joining, he has expanded his operational knowledge of Apache Kafka and Kubernetes, adapted to building relationships in a remote-first workplace, and looks forward to gaining more experience with Flink. He describes moving into a customer-facing advisory role as an opportunity to develop influence and persuasion skills without dictating customer decisions, and he values Confluent’s culture of ownership, collaboration, diversity, empathy, and constructive conflict resolution.

Prescription medication distribution depends on coordinated, accurate data among manufacturers, distributors, providers, pharmacies, and regulators to ensure authorized dispensing, patient safety, and compliance with rules such as the Drug Supply Chain Security Act. The passage argues that batch processing, legacy systems, siloed databases, duplicate records, and point-to-point integrations can delay authorization decisions, increase manual work, and contribute to shipping errors or medication waste. It presents real-time data streaming as a way to maintain a continuously updated compliance matrix containing medication classifications, provider credentials, pharmacy approvals, shipment details, and regulatory requirements. Confluent’s proposed hybrid architecture combines on-premises and cloud deployments, change-data-capture connectors, cluster linking, stream processing with Flink, and data governance tools to create live views of eligible providers, ineligible providers, approved transactions, and delivery status. According to the passage, these capabilities could improve distribution efficiency, reduce compliance risks and fraud, optimize delivery routes, support real-time analytics, and provide a foundation for future AI applications such as interaction monitoring, pharmacist support, and adaptive logistics.