October 2024 Summaries

Semantic data contracts are an alternative to traditional governance APIs for decentralized data management in data mesh architectures. They embed governance directly into data pipelines, defining both structure and meaning, and can be managed as code using tools like GitHub, offering a flexible and adaptable approach to modern data engineering. Semantic data contracts include schema definition, semantic definition, data quality rules, access policies, business rules, and usage docs, allowing for collaboration between business and technical people and enabling safe migrations without obstacles. They aim to empower organizations to maintain high data quality and compliance while sacrificing agility or innovation, and can be used to enable portability of data catalogs and build a portable data lake.

SQL Benchmarking: comparing data pipeline tools` This benchmark compared popular data pipeline tools - Fivetran, Stitch, Airbyte, and dlt (data load tool) - to determine their performance in terms of speed, cost, ease of use, and customizability. The test used the industry-standard TPC-H dataset and evaluated each tool's execution time, cost, and performance tuning capabilities. Key findings included: dlt using ConnectorX was the fastest, with significant cost savings compared to other tools; Airbyte was the slowest; parallelization improved performance for some tools but had diminishing returns due to network and database limitations; and Python performance in EL (e.g., Fivetran) is not a bottleneck. The study highlights the importance of considering these factors when choosing a data pipeline tool.

The path to vendor-agnostic Data Platforms is a journey towards achieving "portability" in databases, which involves making compute components interchangeable and environment agnostic. However, the database ecosystem has diverged from programming languages in terms of standardization, portability, and vendor lock-in. Historically, databases were designed for organizations with a focus on stability, performance, and long-term support, leading to vendor lock-in. The rise of cloud computing further fragmented the landscape. Currently, achieving portable compute requires using open-source technologies like Postgres or Clickhouse, data lakes, lakehouses, Duck Pond, or multi-engine data stacks. Adding an abstraction layer can serve as a standard, enabling technology agnosticism and reducing vendor lock-in. Decoupling extension usage from core SQL access and adopting semantic data contracts can further reduce risk and break vendor locks, igniting competition and potentially leading to increased inter-operability. The portable data lake is a bridge to the future by creating a framework for vendor-agnostic components.

Harness builds an end-to-end data platform using dlt + SQLMesh. Harness relies on dlt to handle data extraction, normalization, and loading of data sources, with 14 active sources and growing. dlt also handles schema automation and evolution. The company uses BigQuery as its data warehouse. Once the data is in the data warehouse, Harness uses SQLMesh to provide data contracts between the data in the data warehouse and how it's being transformed downstream. dlt and SQLMesh are integrated to achieve interoperability between them. Harness empowers its senior data engineer, Alex Butler, to make technical decisions without needing to go to someone else, allowing him to build everything himself. Alex was previously using Singer, Meltano, and dbt, but found dlt to be a more efficient solution. He migrated his company's core SaaS service pipelines to dlt in just 2 weeks and achieved full confidence that the data pipelines would never break due to changing schemas. Harness has integrated dlt into its own platform, allowing users to control what resources of sources are enabled or disabled through the UI. This integration enables "data democracy" for its product, business, and operation teams, allowing them to independently satisfy a majority of their data needs through no-code self-service. Alex also migrated his company's dbt models to sqlmesh, which provided him with column-level lineage and interoperability between dlt and sqlmesh. The integration allows Harness to switch test, development, and production destinations easily, and provides an operational dashboard that assesses the health of individual columns per model in a fully automated way.

The dlt-dbt generator is a tool that leverages metadata from the start to empower data engineers to create scalable and flexible infrastructures for turning raw data into actionable insights with minimal friction. The generator provides a way to standardize and harmonize data before modeling, enabling interoperability and complex governance out of the box. It allows users to capture metadata at ingestion, dimensional modelling before loading, and enables running the modelling on the conformed layer, standardizing and harmonizing the data before modeling.

The modern data stack is fragmented, with tools operating in silos and lacking effective metadata sharing. Open-source standards like `dlt` (Data Load Tool) and SQLMesh aim to bridge these gaps by promoting interoperability and seamless communication among tools. By adhering to open standards, tools can expose internal metadata, reducing the need for manual integrations and glue code. The integration of SQLMesh with dlt enables faster scaffolding, incremental processing support, and several other benefits that enhance data quality and consistency, making it a powerful tool for data platform engineers.

A portable data lake is a development environment that combines the benefits of data ponds, which enable fast local experimentation and development, with the governance and scalability required for production environments. It aims to address the challenges faced by data professionals in setting up local environments for large-scale data work, such as data access, scalability, and governance issues. The proposed solution, a pip installable platform called a portable data lake, integrates features like integrated caching, governed pipelines, unified data access, and fast-track to production, ensuring robust governance while enabling seamless collaboration across teams. It leverages open standards like Delta Lake, Parquet, and Iceberg to provide efficiency gains and unlock new development paradigms in data engineering.

Migrating from Fivetran to dlt involves standardizing data pipelines for organizations prioritizing control, customization, and cost efficiency. While Fivetran offers ease of use and pre-built functionality, dlt provides a code-first customizable solution with robust metadata fields for effective tracking and auditing. By consolidating historical and new data into unified views, teams can ensure a seamless transition without sacrificing data integrity or performance. A similar approach can be applied to other data sources like Salesforce, Shopify, and SQL databases.