March 2023 Summaries

Data contracts have emerged as a significant topic in the data space, serving as agreements between software engineers, who produce the data, and data teams, who consume it. Unlike APIs in software engineering, which are well-established for facilitating reliable communication between systems, data contracts aim to prevent upstream changes from disrupting downstream dependencies by fostering human-to-human agreements. The rise of data contracts is driven by the need to formalize and stabilize data pipelines, especially as data teams have historically been out of the loop with software teams, often resorting to extracting data directly from databases without proper agreements. This practice can lead to untrustworthy data due to the lack of consent and communication between teams. While software teams may be reluctant to prioritize building APIs for data teams due to existing backlogs, it is crucial for both data and software teams to collaborate and align their efforts to achieve overarching business goals. This alignment goes beyond mere API implementation, requiring organizational culture shifts to ensure that data teams are integrated into the broader workflow, ultimately enhancing the business's value through collaboration.

Confluent has achieved a Google Cloud Ready - AlloyDB designation for its integration with AlloyDB for PostgreSQL, Google Cloud's fully managed PostgreSQL-compatible database service. This designation recognizes solutions that have met functional requirements and been validated by Google Cloud engineering teams. The integration allows Confluent's data streaming platform to stream data into AlloyDB and invoke Google Cloud's VertexAI predictions seamlessly, leveraging AI/ML capabilities. As a result of this achievement, customers can confidently use Confluent products with AlloyDB, and both companies will have more opportunities for close collaboration in the future.

Confluent is a data streaming platform that enables organizations to treat data as a continually updating stream of events, creating a central nervous system for the entire organization. This allows companies in the healthcare space to collect and process a continuous flow of data in real-time, enabling the discovery of new insights and maximizing patient centricity. Confluent powers various use cases such as benefits determination, fraud detection, claims processing, and data sharing, which are critical in the healthcare industry. The platform helps organizations create personalized care plans, detect patterns and outliers to flag insurance claims, process claims faster with AI and ML, and enable efficient and secure data sharing between providers, payers, and patients.

Confluent's data streaming platform enables organizations to treat data as a continually updating stream of events, creating a central nervous system and enabling the discovery of new insights in real-time. Confluent powers benefits determination by aggregating data from various touch points, fraud detection by layering on AI and ML technologies, claims processing by decoupling legacy architectures, and data sharing by powering secure and efficient data streaming. This platform is critical for healthcare organizations, allowing them to create a new paradigm that supports collecting and processing continuous data in real-time, maximizing patient centricity and enabling the creation of personalized care plans and outcomes. Confluent's fully managed data streaming platform has transformed from a self-managed Kafka solution, offering insights into modern data architecture.

The convergence of machine learning (ML) and data streaming markets is hindered by socio-technical challenges, but solutions like Apache Kafka offer a promising path forward. The ML field is at an inflection point, with rising investments but limited widespread deployment due to cultural barriers and organizational inertia. Kafka serves as a central nervous system for data-driven organizations, connecting diverse data sources and enabling real-time analytics crucial for ML applications. The synergy between Kafka and ML is evident in its ability to handle event-driven data, reducing latency and optimizing real-time solutions, which is essential for applications like fraud prevention and proactive infrastructure maintenance. However, challenges remain, such as bridging the gap between Java, which dominates data streaming, and Python, the preferred language for ML. Overcoming these barriers requires organizational restructuring towards feature-oriented teams and embracing a Modern Data Flow approach, which includes data mesh and microservices paradigms. As the industry works towards integrating these ecosystems, the potential for innovation and efficiency gains is substantial, though the path is fraught with both technical and cultural hurdles.

Evi Schneider, a longtime Confluent marketer, describes her progression from leading Central European marketing across events, campaigns, partnerships, and localized German-language assets to a European campaign management role focused on scalable programs. She joined when Confluent and Apache Kafka were relatively unknown, drawn by the opportunity to learn and help establish the emerging data-in-motion category. Schneider recounts Confluent’s rapid evolution from an on-premises focus to a cloud-based product, major hyperscaler relationships, an IPO, and a growing customer base that includes Bosch, BMW, Lufthansa, Siemens, Mercedes-Benz, and Swisscom. She highlights the company’s collaborative culture, flexible working practices, and continual learning opportunities, while emphasizing real-time data’s applications in retail customer experiences and fraud protection. Her upcoming work includes the Data in Motion Tour in Frankfurt and Zurich and her new position as EMEA’s first Global Campaigns Manager, and she encourages prospective employees to embrace unfamiliar opportunities and learn through experience.

Confluent promoted its March 30 Data in Motion conference in Washington, D.C., as an event focused on helping industry and government organizations use real-time data streaming to support faster decisions, personalized services, operational routing, security, and field communications. The program was set to cover Apache Kafka’s development and role as a distributed event-streaming platform, along with strategies for evolving Kafka deployments to meet demand for rapid data access. Featured sessions included a keynote by NASA’s Joe Foster on the agency’s data challenges and FedRAMP sponsorship, a Women in Data panel with public-sector and consulting leaders, and hands-on workshops on public-sector streaming pipelines, smart edge computing, data mesh, and cybersecurity use cases. The page also promoted registration, event-tour information, blog subscriptions, and related posts about the Boston Data in Motion Tour and Current 2022.

Stream-processing frameworks such as Kafka Streams, Apache Flink, and Apache Spark generally favor a single primary programming language, forcing developers either to use embedded SQL as a neutral layer or to rely on more complex inter-process communication between language runtimes. The post suggests that GraalVM, particularly its Truffle framework for implementing languages through abstract syntax tree interpreters, could make cross-language stream processing more practical and efficient. If adopted, it could allow developers to use any language supported by Truffle, enable shared language-agnostic profiling, debugging, and monitoring tools, and centralize performance improvements through GraalVM’s just-in-time compiler rather than requiring separate optimization efforts for each runtime.

Confluent’s Data in Motion Tour opened its North American schedule in Boston, where attendees discussed how organizations use data streaming and managed Kafka services to support real-time products, customer experiences, and modernization efforts. Confluent Field CTO William LaForest emphasized that while Apache Kafka is relatively easy to adopt, operating it at production scale creates operational demands involving provisioning, scaling, upgrades, security, monitoring, and rebalancing. Clean Harbors solution architect Manoj Vasudev described choosing Confluent Cloud after finding that self-managed open-source Kafka would create substantial long-term maintenance overhead, while Nasdaq Director of Software Engineering Ruchir Vani said his team is considering a move from open-source Kafka to Confluent to reduce management burdens and devote more engineering capacity to data products. The event also highlighted Confluent Cloud’s managed connectors, scalability, and support for broader internal data access, alongside networking, workshops, demonstrations, and customer discussions planned for future tour stops.

The initial excitement surrounding "Big Data" promised revolutionary solutions across various domains by leveraging massive datasets and processing capabilities, but it eventually succumbed to the typical hype cycle and became less of a buzzword as its practical applications matured. The initial tools like Hadoop and its Hadoop Distributed File System (HDFS) paved the way for more efficient and user-friendly successors like Amazon's S3, which improved upon distributed file systems and became integral to modern cloud storage solutions. MapReduce, which brought computation to data, was replaced by faster alternatives like Apache Spark, which simplified data processing with higher-level abstractions like SQL. Big Data's legacy persists in the way it redefined large-scale computing, emphasizing resilience and recovery as fundamental features in cloud services, while its technologies have become embedded in everyday operations, demonstrating the industry's shift from novelty to necessity.

With Confluent Cloud, Stream Designer enables users to build RAG-enabled GenAI pipelines quickly and easily using a visual canvas. The latest release of Confluent Cloud provides new features that allow users to create and update pipelines with SQL source code by using the Confluent CLI and the Pipelines REST API. This process automatically detects and securely stores passwords and secrets used in connector ksqlDB statements, updating them to use variables that can be updated and managed separately from the SQL code. Users can manage their pipelines programmatically using the Pipelines REST API, which enables secret rotation by using a patch file. With this update, users have more flexibility and control over their pipeline life cycle, allowing them to build streaming data applications with ease.

Jade Bowen describes her progression from Confluent’s 11th ANZ employee and a New Zealand enterprise account executive serving one customer to senior manager for ANZ customer success, overseeing a team and a regional base of roughly 20 clients. Her technology career began with interests in coding and was shaped by roles in software licensing, data centers, Gartner consulting, and Microsoft Azure engineering before she joined Confluent because of its data-streaming mission. She argues that real-time data helps organizations respond to economic, environmental, and social change, citing customers that use stream processing to tailor global services and develop region-specific products, as well as a public-sector initiative using accurate real-time data to improve road safety. Bowen also highlights Confluent’s remote-first culture, wellness benefits, employee communities, and emphasis on opportunities for women and minorities, while encouraging creative and ambitious candidates from varied backgrounds to consider careers with the company.

The text discusses the limitations of traditional centralized and batch-based data management systems and the shift towards modern paradigms like data mesh and event-driven architecture (EDA) to better handle data management in organizations. It emphasizes managing data as a product, with clear ownership, defined scopes, and interfaces, facilitated by a self-serve platform to ensure federated governance and interoperability. The article explains how data products can be automated throughout their lifecycle using descriptor files, specifically through the Data Product Descriptor Specification (DPDS) and the Open Data Mesh (ODM) Platform. An example from the logistics sector illustrates how to manage a stream application as a data product, with Confluent Cloud services enabling the creation and management of these products. The text highlights the importance of decentralization and real-time data processing in overcoming the complexities of current data management challenges, advocating for a combination of data mesh and EDA strategies to optimize data handling and innovation.

Using John Carpenter’s The Thing as a metaphor, the piece describes how data teams face persistent uncertainty when large, complex datasets and opaque machine-learning models make it difficult to distinguish reliable insights from flawed ones. Errors in source data, statistical methods, or software can go unnoticed at terabyte or petabyte scale, potentially leading to harmful business decisions and undermining confidence in all analytics. It argues that trust must be the foundation of automated insight, requiring safeguards such as data observability, contracts, testing, clearer abstractions, and more mature data practices. As analytics and algorithmic automation become increasingly influential, preventing and detecting failures before they reach production is presented as a central industry challenge for 2023 and beyond.