March 2024 Summaries

Smriti Sridhar, a Solutions Architect in Confluent’s Professional Services organization, joined the company through its Associate Consulting Engineering program after earning a master’s degree in systems engineering and seeking an alternative to a conventional software development career. Nearly three years later, she leads Professional Services efforts for Confluent’s Women’s Inclusion Network, creating informal forums and initiatives centered on increasing women’s visibility through conferences and technical presentations, providing education and mentorship through shared experience, and supporting retention and career growth. She credits supportive peers and managers, a culture that treats mistakes as learning opportunities, and access to new challenges with motivating her to remain at Confluent despite initially seeing relatively few women in her field. Sridhar encourages women in technology to ask questions, seek help, take on unfamiliar opportunities, and recognize that roles in Professional Services can open paths into areas such as product management, solutions engineering, and technical sales.

The text discusses using Apache Kafka to track events in a large codebase, specifically GitHub's data sources (REST + GraphQL APIs). It explains how to use the Confluent GitHub source connector to get GitHub events into a Kafka topic and then process those events using Kafka Streams topology. The author also provides an overview of data pipelines, sources, and sinks, as well as details on implementing a state store in Kafka Streams. Furthermore, the text touches upon extending the project by adding a sink and mentions other resources for learning more about Kafka demos, Flink SQL tutorials, and resolving "unknown magic byte" errors.

Hands-on Flink Workshop: Implement Stream Processing | Register Now. The estimated cost of network downtime for telecommunication companies can exceed billions of dollars due to equipment failures, software misconfiguration, and power outages. To avoid these costs, telcos need a holistic view of their networks to proactively identify and resolve issues. A data streaming platform can transform how telcos monitor and address network health and support issues by ingesting and processing real-time data at scale from customer behavioral data, network performance metrics, and subnetwork data. This approach enables predictive customer support, ensuring SLAs, saving time and cost, improving resource allocation, accelerating new feature rollout, increasing trust and transparency, and reducing churn. However, telcos face technical challenges such as siloed data, unprecedented volume of data, disconnected teams, batch ETL/ELT data pipelines, legacy technologies, lack of scalability in running after-the-fact jobs, and lack of visibility into other teams and shared data views. Confluent's data streaming platform can overcome these challenges by analyzing real-time data holistically, training predictive algorithms to identify patterns indicative of potential problems, enabling swift intervention to avoid widespread disruptions.

Data mesh is presented as a decentralized approach to overcoming the growing complexity of fragmented operational and analytical data systems by treating domain-owned data as discoverable, governed, reusable products. Built on four principles—domain-driven ownership, data as a product, federated computational governance, and self-service platforms—it aims to help organizations provide trustworthy, secure, real-time data across teams without relying on centralized control. The Confluent Data Streaming Platform is positioned as an implementation foundation through managed connectors, stream processing, governance tools, and a data portal that support the creation, enrichment, discovery, and sharing of live data products across hybrid and multicloud environments. The accompanying ebook describes a five-stage adoption model, organizational and technical practices for broad data mesh adoption, and customer examples such as automotive logistics firm ACERTUS, which uses streaming data to package customer, vehicle, and transit information into analytical products.

Amy Koh, a Senior Solutions Architect in Confluent’s Professional Services organization, works with customers to design and implement streaming data architectures that address varied business challenges. Inspired by Confluent co-founder Jay Kreps’s book I Heart Logs, she views data in motion as central to digital innovation and helps organizations use Confluent solutions for applications ranging from fintech payment, lending, compliance, and risk systems to real-time modernization in the music industry. Koh says her motivation comes from influencing digital transformation, collaborating with knowledgeable colleagues, and receiving support for professional development through enablement sessions, subject-matter-expert forums, and opportunities to explore new responsibilities. She values direct customer engagement for the insight it provides into industry trends and product improvement opportunities. Koh also highlights Confluent’s remote-first culture, which allowed her to work from Asia while spending time with family, and its diversity and inclusion efforts through employee resource groups such as ACON and WIN, along with cultural events.

Kafka Summit London 2024 brought together more than 3,500 in-person and online participants to celebrate Apache Kafka’s progress, including the project’s 1,000th Kafka Improvement Proposal, and to discuss the growing role of data streaming in large-scale operational systems. Confluent announced the general availability of Confluent Cloud for Apache Flink, introduced the early-access Tableflow feature for converting Kafka topics into Apache Iceberg tables, and reported that its Kora cloud-native Kafka engine is now 16 times faster than open-source Kafka. Additional announcements included expanded managed connector capabilities, default-enabled Stream Governance with broader regional coverage and uptime commitments, and Twinlabs.ai’s win in Confluent’s $1 million Data Streaming Startup Challenge. Sessions covered code-first Flink development, checkpointing for reliable stream-processing applications, hands-on customer-data pipelines, approaches to querying streaming systems, real-time generative AI, and Kafka resilience through health checking. The event concluded with plans for future community gatherings at Kafka Summit Bangalore and Current 2024.

Confluent named London-based startup TwinLabs.ai the winner of its 2024 Data Streaming Startup Challenge, awarding it a $500,000 seed investment for its AI-powered, scalable real-time digital twin platform built with Confluent Cloud and Apache Kafka. TwinLabs.ai uses streaming data from IoT sensors and business systems to continuously monitor and analyze complex events and environments, drawing on experience developing digital twins for the Tour de France. The global competition received nearly 100 applications from 22 countries, which were narrowed to finalists Atomic Tessellator, Busie, and TwinLabs.ai through evaluations by Confluent leaders and venture capital investors. The challenge highlighted the expanding role of real-time data streaming, processing, and analytics in enabling new products and industry applications.

Confluent Schema Registry is an essential tool for managing schemas in data streaming platforms like Apache Kafka, facilitating the interoperability of producers and consumers by leveraging schema IDs. It supports schema evolution through subjects and versions, ensuring backward compatibility to allow consumers to read older messages. Data contracts are crucial for defining data structure, integrity constraints, and metadata, while schema normalization aids in treating syntactically different schemas as identical. Pre-registering schemas is recommended for production environments via CI/CD pipelines to ensure controlled deployment. The topic-based subject name strategy is preferred, as it efficiently manages schema usage in Kafka topics, while schema composition with references promotes reuse and adherence to the DRY principle. Compatibility levels, with a focus on backward compatibility, play a vital role in ensuring smooth data evolution, whereas schema migration rules offer flexible handling of complex schema changes. Schema deletion is generally discouraged due to the immutable nature of schemas, although soft and hard deletions are possible under certain circumstances. Overall, Confluent Schema Registry provides robust tools and practices for managing schema lifecycle and ensuring data quality through rules and contracts.

Confluent Cloud for Apache Flink has reached general availability across all three major cloud service providers, providing customers with a fully managed and serverless stream processing experience. This unified platform combines the performance and rich feature set of Apache Flink with the scalability and reliability of Kafka, enabling users to process data in real-time and create high-quality streams to power GenAI applications. With features such as SQL Workspaces, Topic Actions, Terraform support, and expansion into GCP and Azure, Confluent Cloud for Apache Flink offers best-in-class stream processing capabilities, reliability, and performance, empowering users to harness the full potential of streaming data and accelerate their time-to-market and business outcomes.

The introduction of Tableflow by Confluent aims to unify the operational and analytical estates in organizations by making it easy to feed Apache Kafka data directly into data lakes, warehouses, or analytics engines as Apache Iceberg tables, removing the need for duplicative work and reducing complexity and cost. This is achieved through innovations in Confluent's Kora Storage Layer and a new metadata materializer that handles schema mapping, schema evolution, and type conversions, ensuring seamless integration with popular catalog services such as AWS Glue and Polaris Catalog. By unifying batch and stream processing, Tableflow simplifies data infrastructure, reducing complexity and cost, and provides a convenient way to get data into platforms built around Iceberg.

Confluent Cloud has announced its latest updates in 2024, highlighting the addition of serverless Apache Flink capabilities and GenAI applications. The platform aims to enhance data-driven decision-making and innovation through improved stream processing, real-time data processing for AI applications, and secure networking paths with DNS forwarding and Egress Access Points. Additionally, Confluent Cloud has expanded its connector portfolio, offering over 80 fully managed connectors with new pricing updates, reduced data transfer costs, and improved usability. The platform also introduced Stream Governance enhancements, including regional coverage expansion, improved SLAs, and a new Advanced package for Schema Registry. Furthermore, Confluent Cloud has launched the Connect with Confluntion partner program, enabling more real-time use cases and innovation through integrations with technology partners.

Apache Flink 1.19 marks another step forward in stream processing technology with numerous new features and enhancements aimed at enhancing the system's reliability and flexibility, while setting the groundwork for the upcoming Flink 2.0. The release focuses on significant Flink Improvement Proposals (FLIPs) and other contributions, demonstrating how Flink contributes to advancing stream processing and supporting the development of more dynamic, efficient, and user-focused data streaming applications. Key features include improved configuration processes, enhanced support for batch jobs, and advancements in SQL and Table API capabilities. The introduction of dynamic source parallelism inference, async scalar functions, and a built-in Java profiler further improve performance and usability. Additionally, Flink 1.19 introduces beta support for Java 21 and serves as a foundational step toward the release of Flink 2.0.

Bahar Sandal, a Senior Account Executive for EMEA at Confluent with more than 13 years of technology experience at companies including Microsoft and Salesforce, describes how her technical background and Confluent’s customer-focused environment support her work on digital transformation, enterprise sales, and international strategy. She is a founding member of Girls in Tech Switzerland and a board member of Women in Tech Switzerland, where she has helped lead initiatives such as a hackathon for refugees and women facing immigration challenges and a STEM Exploration Week reaching 20,000 students. Sandal identifies Confluent’s customer-centric culture, teamwork, inclusion, employee resource groups, training, and leadership support for diversity, equity, and inclusion as important aspects of the company. Looking ahead, she plans to deepen customer relationships, deliver technology solutions for evolving business needs, and expand awareness of Confluent’s offerings across Switzerland.

Smart warehousing represents a significant shift in retail logistics, utilizing technologies such as IoT, robotics, cloud-native platforms, machine learning, and data streaming to enhance efficiency and accuracy in inventory management. By 2027, over 75% of companies are expected to adopt such automation, according to Gartner. This transformation enables features like real-time inventory tracking, predictive maintenance, and optimized stocking algorithms, which improve order fulfillment and reduce waste. However, challenges such as complex integration tasks, outdated batch processing systems, and data silos present obstacles. Platforms like Confluent offer solutions by providing real-time data streaming, which supports automation, reduces errors, and improves resource allocation through connected data environments. This shift not only leads to cost savings and increased productivity but also supports more agile and sustainable operations, allowing retailers to meet growing consumer demands for rapid delivery and efficient service.

Victoria Mileham and Murielle de Gruchy reflect on women’s experiences in technology, emphasizing empathy as a potential leadership strength, the value of diverse personal support networks, and the importance of advocacy from both women and male allies. They describe Confluent’s mentorship culture and Women Inclusion Network as mechanisms for amplifying women’s voices and career development, while noting increased representation of women in senior roles through hiring and promotion. Although they see progress in workplace equality, remote work, and organizational awareness, they acknowledge that the broader technology sector remains male-dominated, particularly in engineering and sales, and argue that companies need women represented across all functions to support innovation and effective business strategy.

Confluent’s 2024 Data in Motion Tour stop in Washington, D.C., scheduled for March 21 at the Westin Tysons Corner, brings government, industry, partners, and customers together to discuss practical data-streaming applications and organizational adoption. Building on the prior Public Sector Summit, the event features a keynote by retired U.S. Army Colonel Andrew Hall, whose experience spans federal cybersecurity leadership and academic work in cybersecurity and data science. A Women in Data panel with leaders from Capital One, ASRC Federal, and Deloitte will address career development, data transformation initiatives, and the future of women in technology. Attendees can also join technical sessions on developing streaming data products with Confluent’s recent innovations and a hands-on AWS and MongoDB lab focused on building an end-to-end hybrid data-streaming pipeline.

Generative AI adoption expanded rapidly in 2023, but organizations seeking to operationalize it face persistent challenges from siloed, batch-oriented, and difficult-to-access enterprise data. While traditional machine learning often required costly custom models trained on centralized data lakes, generative AI increasingly relies on pre-trained large language models supplemented through retrieval-augmented generation with current proprietary business information. Real-time data is particularly important for applications such as customer-service chatbots, which require up-to-date account, inventory, pricing, and operational details to provide useful responses. The passage argues that data-streaming platforms can address these needs by connecting, processing, governing, and delivering enterprise data continuously as reusable data products, enabling reliable inputs for AI and ML systems. It identifies real-time analytics, low latency, security, and governance as major reasons organizations invest in streaming, notes Kafka’s broad enterprise use, and presents Confluent as a platform designed to unify data across systems. Despite substantial interest, the passage says relatively few organizations currently use streaming data effectively for AI or ML, leaving significant room for adoption as AI initiatives mature.

Leandro Galli, a Senior Solutions Engineer at Confluent, describes his career from beginning programming in Brazil at age 12 through nearly two decades in telecommunications, international consulting based in Spain, and his eventual move to Confluent in 2020 after seeing Apache Kafka’s growing impact on customers. Initially the sole solutions engineer supporting fast-growing digital-native companies across EMEA, he helped during a period of rapid team expansion before relocating to Dubai in 2022 to work with larger enterprise customers in the Middle East. His role centers on assessing data-streaming needs, supporting customer implementations such as real-time fraud detection and architecture modernization, and collaborating with marketing, partners, product managers, and engineers. Galli attributes his interest in Confluent to its data-streaming technology, its potential to help businesses beyond telecommunications, and its people, while describing the company culture as collaborative and guided by values of humility, empathy, and learning. He also identifies a scholarship that enabled him to study in France as a pivotal career decision, leading to international experience, multilingual skills, and work across multicultural teams in Europe and the Middle East.