March 2025 Summaries

The author of the article tested the capabilities of large language models (LLMs) in generating pipeline code for the Pipedrive API, specifically focusing on feature extraction, pipeline code generation, and memory-based intuition. The tests revealed that relying solely on LLMs' memory and intuition is unrealistic and that documentation quality and structure significantly impact the accuracy of feature extraction. The author also developed a structured extraction prompt to evaluate the feasibility of generating pipelines from APIs and identified key issues with Pipedrive's API documentation, including authentication and response formats. To overcome these limitations, the author suggests using partially-built pipelines and inspecting responses to gather missing information. The article concludes that establishing a definitive benchmark for AI-generated data pipelines is necessary to improve their accuracy and reliability.

The Write. Audit. Publish (WAP) framework is a software engineering discipline applied to data engineering, aiming to ensure the quality and trustworthiness of data before it reaches production. This approach involves writing data in isolation, auditing for correctness, quality, and compliance, and publishing only validated data. A hypothetical scenario using the dlt dataset tool demonstrates how WAP can be implemented to catch errors upstream, preventing breakages and ensuring trustworthy data. By following this pattern, data teams can prevent bad data from reaching production, save time on last-minute fixes, and gain confidence in their data pipeline.

The author, Adrian Brudaru, shares his experience of rewriting an old Python script to load data from the Pipedrive API using the dlt and Cursor tools. The original script was complex, unmaintainable, and lacked modern best practices, making it difficult to hand over to a team. By using dlt and Cursor, Adrian created a declarative, self-documenting, and easily maintainable REST API source with explicit schema management strategy, resulting in a cleaner, more modular pipeline that can be scaled and tested. The author highlights the importance of data contracts, modularity, reusability, and observability, and encourages readers to modernize their legacy ETL scripts using dlt + Cursor.

The author of this text is exploring the use of Large Language Models (LLMs) to generate data pipelines, specifically for connecting to REST APIs. The problem they face is that LLMs struggle with extracting key parameters such as pagination, authentication, and primary keys from API documentation, which are often missing or not easily inferable. To overcome this challenge, the author uses an Airbyte YAML source as a starting point and feeds it into a Cursor project configured with custom LLM prompts and documentation context. The LLM is then used to convert the YAML into a Python pipeline using dlt's REST API source. The results show that the LLM can successfully generate a working pipeline, including incremental loading and authentication, on a simple example API. Key learnings from this experiment include the importance of preparing documentation in an LLM-friendly format, custom prompts, and instructions for better outcomes, as well as recognizing that some information may still require human intervention.

Iceberg, a technology that's gaining traction in the data engineering community, is being hailed as a revolution due to its ability to address many of the pain points associated with traditional data lakes. It offers ACID transactions, schema evolution that works, and a table format that doesn't lock users into a single vendor. This allows companies like Netflix, Apple, and Adobe to bet on Iceberg early. The technology is also being used by Trino, Snowflake, and BigQuery, further solidifying its position as an inevitable choice for data engineers. By decoupling compute from storage, Iceberg enables AI workloads to run on lightweight engines like DuckDB and Trino, reducing costs and improving efficiency. Additionally, Iceberg provides a structured, versioned memory that ensures AI systems retrieve consistent, historical data for reproducibility and reinforcement learning. With the rise of machine learning and AI, which has forced data to evolve, Iceberg is well-positioned to reshape data engineering by providing a composable, open, and interoperable solution.

Data engineering, despite its importance, often operates with a "YOLO" (You Only Live Once) approach, where transformations are pushed straight to production without thorough testing. This can lead to wasted compute, slow iteration, pipeline lockups, and high costs of fixing defects. In contrast, CI/CD practices in software engineering ensure that every change is tested, integrated, and deployed in a structured, automated way. The article argues that data engineers need the same level of rigor as software developers to address the unpredictability of their dynamic environments. To bridge this gap, tools like dlt+ Cache aim to bring structured, repeatable testing to data engineering, enabling transformations to be tested locally before deployment and ensuring schema changes are validated instantly.

The Continue Hub has released the initial building blocks and assistants to work towards creating a data infrastructure for trustworthy data engineering AI. The dlt Assistant helps users develop, debug, and inspect their dlt data pipelines, while the dlt+ Assistant allows teams to interact with dlt+ Project, destinations, and pipelines in a standardized way via a YAML Project manifest. The assistants are built on top of the dlt construct of pipelines, which connect sources to resources and destinations. The Continue Hub aims to work with users as well as the wider community to develop iterative workflows for debugging and testing dlt pipelines. The assistants can be integrated into AI-enabled tools and platforms, such as VSCode or Jetbrains, and provide a more encompassing understanding of project or data platform. The team is seeking feedback from users and companies with data engineering teams that embrace AI code editors to learn about their needs and workflows.