February 2025 Summaries

We've partnered with Continue to launch custom AI code assistants for data engineering, building on our foundation of dlt and dlt+ pipelines. This partnership aims to create a more efficient and trustworthy data engineering workflow by leveraging the Continue Hub's models, context, and other building blocks. Our goal is to enable developers to amplify their work, rather than automating it, as we believe that human engineering time will increase in monitoring, maintaining, and improving AI software development systems. We're working towards a future where compound systems can generate trusted data, unlocking additional data engineering assistants and building blocks. Our focus includes developing custom AI code assistants that integrate with developers' existing workflows, utilizing the Anthropic MCP standard for reliable context, and supporting emerging standards like Iceberg Apache Iceberg for large datasets. By doing so, we aim to address the lack of trust in data, which has been a major factor in enterprise deployment of AI. We invite companies and developers to join us in this community-driven effort to solve some of the problems together.

The developer framework dlt+ has been released in early access, featuring two initial features: the dlt+ Project and dlt+ Cache. The dlt+ Project is a declarative yaml collaboration point that enables easy onboarding and collaboration with non-Python developers, while also providing a single, authoritative manifest for pipeline definition, deployment, and orchestration. The dlt+ Cache is a database-like portable compute layer that allows users to shift parts of their data workflows earlier in the development process, enabling faster iteration, safer quality checks, and vendor-agnostic transformations. Early testers can collaborate with the developers to shape the product further, and more features are planned for release in subsequent weeks.

The dlt+ Project is an open-source tool designed to solve the challenges faced by data teams when building and maintaining pipelines. It provides a declarative configuration layer through a single manifest file, enabling teams to work faster, smarter, and with less friction. The project is built for teams managing data pipelines at scale, ideal for data engineers, analysts, and organizations requiring seamless multi-environment support. With dlt+ Project, teams can define their pipelines in YAML, collaborate without deep coding expertise, and achieve seamless dev-to-prod transitions. By providing a single source of truth for pipeline configuration, transformation logic, and environment settings, the project reduces dev-to-prod discrepancies, onboarding delays, and collaboration bottlenecks, ultimately streamlining data team development and collaboration.

The dlt+ Cache is a portable compute layer for developing, testing, and validating data transformations before they hit production. It provides a staging layer that allows engineers to test, validate, and debug transformations without running everything in the warehouse. The cache offers an engine-agnostic interface, supports various SQL dialects, and integrates with existing data workflows. It enables users to run transformations locally, validate schema changes, and test without burning cloud costs. The dlt+ Cache is designed to solve the problem of debugging transformations in production, which can be time-consuming and costly. By using the cache, engineers can build pipelines more efficiently and reduce the need for dev environments.

Aman Gupta's blog post presents 11 Pythonic data quality recipes designed to enhance data integrity in data pipelines using the Data Load Tool (dlt) framework. Each recipe addresses specific data quality challenges, such as using Pydantic for early data validation, implementing schema freezes to maintain consistent downstream schemas, and utilizing bad data filters and silent value cleaners to handle API drift and unexpected data changes. The post also highlights methods like primary key deduplication to maintain data uniqueness, schema evolution tracking for compliance, and dynamic schema contracts for adaptable quality rules. Additionally, it discusses the importance of schema evolution and contract violation alerts to detect and prevent unauthorized data changes in real-time. These recipes aim to provide practical solutions to improve data quality and trustworthiness as data and team requirements evolve, encouraging readers to experiment with these techniques in their pipelines for cleaner and more reliable data management.

The author of the text argues that the shift from ETL (Extract, Transform, Load) to ELT (Extract, Load, Transform) in data warehousing has created a bottleneck. While ELT made prototyping easier, it's hard to maintain and scale, leading to repetitive work, debugging costs skyrocketing, and companies paying more for less value from their most skilled people. The author proposes introducing a staging layer for data transformations, which would allow engineers to run transformations locally, validate before loading, and test without burning cloud costs. This approach aims to stop prototyping and start building for production, making it production-ready, and ensuring that software engineers don't deploy untested code while data engineers shouldn't deploy untested pipelines.

Debezium is a powerful open-source platform for change data capture that monitors database transaction logs and produces change event streams, indicating inserts, updates, and deletes. It can be used in conjunction with Python to build a robust and scalable CDC solution. The pydbzengine library provides a convenient way to interact with the Debezium embedded engine, simplifying the process of configuring and running Debezium within a Python application. The dlt pipeline is another tool that can be used to load data from Debezium into a destination such as DuckDB. This example demonstrates how to use Debezium and pydbzengine to build a CDC pipeline in Python, capturing change data from a PostgreSQL database and loading it into a DuckDB database using the Data Load Tool (dlt). The code includes explanations of the key components involved, including Debezium, pydbzengine, dlt, DuckDB, and Testcontainers. It also provides a testable example that can be used to experiment with different configurations and workflows. Overall, this project is designed to provide a simple and efficient way to capture change data from databases and load it into data warehouses using Python.