June 2022 Summaries

Datafold announced the discontinuation of active support for its open-source tool, data-diff, as of May 17, 2024, but offers a free trial of Datafold Cloud for ongoing in-database and cross-database diffing needs. Data-diff was launched as an open-source solution designed to automate data quality checks during data replication and migration, addressing common issues such as message loss and inconsistencies due to hard deletes or stale update timestamps. Traditional methods of validating data consistency across databases, such as comparing row counts or hashing, are either too crude or inefficient; data-diff offers a more effective solution by using a binary search-like algorithm on hashes, which enhances both speed and accuracy by only transferring hash values over the network. This method is both IO efficient and highly parallelizable, allowing users to efficiently compare large datasets across different databases. Despite the tool's discontinuation in active development, it remains a significant contribution to addressing data replication challenges faced by many data teams.

Datafold is enhancing the productivity and quality of life for analytics engineers by launching a free-tier and transparent cloud pricing, making important developer tools, such as Data Diff and column-level lineage, accessible to all data teams. These tools address significant gaps in the analytics engineering workflow by providing essential features like metadata and lineage visibility, data diffing, metadata search, data profiling, and monitoring/alerting capabilities. The platform aims to improve the developer experience by offering tools that automate testing and provide insights into data transformations, which have been limited in the past by inadequate tooling. Datafold's new pricing structure includes a free tier for small teams using a modern data stack, a cloud tier with scalable pricing starting at $799 per month, and an enterprise tier for larger organizations with specific needs, ensuring that data engineering tools are available to a wider range of companies regardless of size.