May 2024 Summaries

Foundational has expanded its support for schema management frameworks by announcing official support for Liquibase and Active Record, alongside the existing support for SQLAlchemy. This development aims to bridge the gap between engineering and data teams by providing tools to track upstream schema changes and understand their downstream impact, thereby preventing data incidents. Liquibase, an open-source framework for managing database schema changes, and Ruby's Active Record, a widely used ORM for Ruby on Rails, are now integrated to help users manage and monitor schemas and transactions across major databases like MySQL and Postgres. Foundational's capability to extract accurate data lineage from source code enables automated data contract enforcement through customizable policies, ensuring that changes are safely deployed. These integrations enhance the ability to validate pull requests and track end-to-end data lineage, from operational databases to dashboards, while fostering improved communication and collaboration between teams.

Data lineage, essential for tracking the flow and transformation of data across platforms, poses significant challenges due to the diversity of systems, increasing query complexity, and the substantial effort required for setup and maintenance. Automated data lineage solutions, particularly for cloud data warehouses like Snowflake and BigQuery, have become more accessible, leveraging SQL parsing and query logs. However, these solutions often struggle with coverage beyond warehouses, including BI tools and upstream data sources. Spark-based Lakehouses present additional difficulties due to the complexity of parsing languages like Scala, Python, and Java. Source-code-based data lineage, facilitated by tools like dbt and Databricks, offers advantages such as minimal lag by directly parsing code repositories hosted on platforms like GitHub. The OpenLineage standard enhances interoperability between data tools by standardizing lineage information exchange, despite requiring code changes and adoption within organizations. Effective evaluation of data lineage solutions should consider coverage, setup and maintenance effort, upstream pipeline coverage, and actionable use cases to improve data trust and operational efficiency.

Object-Relational Mapping (ORM) frameworks allow developers to interact with databases using an object-oriented paradigm, with SQLAlchemy being a prominent example for Python that facilitates the translation of Python objects into relational database schemas. The use of ORMs can streamline data modeling and improve maintainability, but schema changes can inadvertently impact downstream systems, particularly when there is a lack of visibility across different teams and tools. Foundational addresses these challenges by integrating with SQLAlchemy to provide automated data lineage and impact analysis, allowing developers to understand and manage schema changes effectively. Unlike Alembic, which is used for managing database migrations, SQLAlchemy serves as the primary source for database schemas, ensuring that all changes are captured and analyzed. Foundational's solution operates within a Git environment, making it easier for teams to implement data contracts and validate changes before they affect live data systems.

dbt is a powerful framework that has become widely adopted by data organizations for building and managing data pipelines, yet as projects grow, certain challenges and limitations have emerged, such as increased warehouse costs and complexities with incremental models. Incremental models, which allow processing only new data, have introduced the challenge of ensuring timely refreshes, especially when changes affect upstream dependencies. Understanding both table-level and column-level lineage is crucial to efficiently manage these updates and avoid unnecessary full refreshes, which can be costly and time-consuming. Schema changes, particularly with incremental models, can lead to overlooked issues, emphasizing the importance of managing dependencies across multiple environments, especially when dbt projects interface with other tools like BI solutions. While these challenges are not inherently complex, they highlight the potential for simple code changes to cause significant disruptions, stressing the need for automated checks and technology solutions to enhance data quality and team productivity.