October 2022 Summaries

Datafold has announced its partnership with Snowflake, achieving Select tier partner status, which will enable it to provide enhanced data quality solutions to joint customers. This collaboration aims to bolster trust and reliability in data used by organizations, allowing them to make more informed and successful decisions. Datafold focuses on a proactive approach to data quality, aiming to identify and address issues before they affect production data pipelines. The partnership is strategic for Datafold as Snowflake serves as a central component in data stacks, allowing organizations to perform various data-related activities such as creating dashboards, conducting data science, building machine learning models, and operationalizing data. Both companies are optimistic that this partnership will empower analytics engineers with the necessary tools to achieve superior data quality.

The text explores the intricate dynamics between people, processes, and technology within data development, emphasizing the critical role of empathy in understanding various personas involved in the data lifecycle. It highlights that while anyone can learn technical skills like coding, navigating interpersonal complexities requires a different skill set. The article discusses how software engineers and operations teams, although primarily focused on solving their own technical problems, inadvertently produce data used for analytical purposes, often leading to a disconnect between data producers and consumers. It underscores the importance of collaborative systems to bridge this gap, allowing data practitioners to effectively utilize data for analytics despite its original intent. Additionally, the text examines how executives and various organizational teams like Ops and Product teams leverage data for decision-making and experimentation, stressing the responsibility of data practitioners to facilitate efficient data use across the company.

Convoy, a digital freight technology start-up, emphasizes the importance of data quality and integrity by implementing Data Contracts, which are agreements between software engineers and data consumers to ensure high-quality and trusted data. These contracts address previous issues of data accountability and clarity by defining clear ownership and facilitating collaboration, allowing data to be treated with the same rigor as APIs. Metaplane, likened to Datadog for data, enhances data observability, providing real-time monitoring and alerting to maintain data quality and trust. Virgin Media O2's Vortex Engine standardizes real-time data sharing through Google Cloud Platform, employing a structured process for data ingestion and processing, ensuring scalability and efficiency. Doctolib, a healthcare teleconsultation service, focuses on minimizing code errors and database conflicts in its data warehouse, using a CI pipeline for automated testing to maintain a high velocity of production changes while ensuring data integrity.

In an episode of the "Monday Morning Data Chat" podcast titled "Improving the Modern Data Stack," hosts Matt Housley and Joe Reis discuss with Gleb Mezhanskiy, CEO of Datafold, the challenges and solutions in data management and engineering. They highlight common issues such as applying data to business logic, change management, and data quality across different company sizes, emphasizing that smaller companies struggle with budgets and expertise while larger ones face political challenges. The conversation explores how technology and collaboration can reconcile differing data interpretations and improve the modern data stack, with practices from software engineering and DevOps being gradually adopted in data engineering. They also discuss the potential and challenges of data contracts, noting the difficulty in establishing cross-tool agreements and the importance of robust change management practices to address data inconsistencies. Ultimately, the focus is on enhancing workflows, understanding metric changes, and ensuring effective automation before implementing model contracts.