May 2026 Summaries

dltHub Transformations, now in public preview as part of dltHub Pro, revolutionizes data processing by converting raw data into clean, usable tables for businesses and agents, drastically reducing the time and resources traditionally required for data migrations and modeling. The tool is designed for a paradigm where agents now write the majority of data pipelines, shifting data work from team-based efforts to tasks manageable by individuals or small teams using the transformation toolkit with platforms like Claude, Codex, or Cursor. This toolkit automates the creation of taxonomies, ontologies, canonical data models (CDM), and Python transformation code, making it accessible for mid-level engineers to handle tasks typically reserved for senior engineers. As demonstrated by early adopters like Navit and Tasman Analytics, this approach enables faster, more cost-effective data transformations, allowing businesses to maintain high service levels without extensive hiring or consultancy engagements. Additionally, the dltHub Transformation engine, utilizing Python and SQL, ensures seamless integration across different data warehouse environments, addressing the growing need for adaptable, schema-aware transformation layers that can keep up with the rapidly increasing number of agent-written pipelines.

dltHub Transformations offers a comprehensive solution to the fragmentation seen in traditional data processing stacks by integrating ingestion, transformation, lineage, and verification within a single execution context. This approach enables a large language model (LLM) to operate with business understanding akin to a senior analyst, facilitated by a clean canonical model, a business ontology, and seamless metadata flow. Unlike traditional setups where context is often lost across separate tools, dltHub ensures continuity, thus allowing agents to perform tasks typically requiring human judgment. This unified approach, exemplified in Navit's experience, improves efficiency, reduces tech debt, and maintains high service level agreements (SLAs) without additional hires by relying on ontology-driven transformations and a consistent runtime environment. The framework supports various platforms, including Snowflake and BigQuery, ensuring adaptability and ease of deployment.

The blog post explores the efficiency of using dlthub for data movement tasks by examining how compute hours translate into data moved across different bottlenecks in data pipelines. It focuses initially on the performance of SQL copy operations, showing that under optimal conditions, dlthub can move up to 65 GB or approximately 350 million rows of Postgres data to BigQuery in one hour, when source and destination are co-located in the same region. The post outlines plans to benchmark additional scenarios involving REST APIs, JSON files, and Parquet files to provide a comprehensive understanding of different bottlenecks. It highlights that most dlthub pipelines face challenges with REST APIs due to rate limits and with JSON files due to high CPU usage for schema inference. The article also provides cost estimates for typical data operations, suggesting that the monthly expenses for data movement are generally modest. Furthermore, a trial version of dlthub is available for potential users to test its capabilities.

In a detailed account of migrating from HubSpot to Attio in just two weeks, the post emphasizes the pivotal role of a strategic workflow rather than relying solely on AI. With one working student, a stakeholder call, and the use of dltHub's agentic transformations, the team navigated the complexities of CRM migration, which involves more than just transferring data rows but also preserving valuable organizational knowledge embedded in sales notes and pipelines. The article highlights the necessity of contextual understanding, which AI alone lacks, in handling legal constraints, schema nuances, and business realities. It outlines a methodical approach involving setting up a canonical data model (CDM), building an ontology from stakeholder meeting transcripts, using a transformation toolkit, and ensuring thorough testing and controlled rollouts. The workflow, bolstered by guardrails like small, manageable pull requests and regression testing, enabled a fast yet careful transition, underscoring the need for human oversight in code review despite AI's rapid capabilities.

dltHub Pro is a new platform designed to make data engineering accessible for Python developers by enabling agents to build and deploy data pipelines using a Claude/Codex/Cursor-native approach. This innovation allows developers, analysts, and AI enthusiasts to independently create and manage data workflows, significantly reducing reliance on specialized data teams. The platform supports rapid data ingestion and transformation, allowing users to build dashboards and analyses quickly from various data sources. dltHub Pro capitalizes on the growing trend of agents writing data pipelines, with the community production of these pipelines increasing from 2,400 to 81,000 per month within a year. The platform offers a modular, Python-first architecture with building blocks like DuckDB and Marimo, ensuring ease of use for individuals with varying technical expertise. As more organizations adopt it, dltHub Pro plans to expand its capabilities with dltHub Scale and dltHub Enterprise versions, catering to larger teams and more complex data engineering needs. The platform is available at a subscription cost, offering credits for managed infrastructure and supporting users with migration services if needed.

dltHub is a newly launched platform designed to make data engineering more accessible to Python developers by leveraging agents to build and manage data pipelines. The platform, which is Claude/Codex/Cursor-native, allows developers and analysts to quickly ingest and transform data without waiting for dedicated data teams, enabling faster decision-making and reducing bottlenecks in analytics workflows. dltHub supports creating pipelines locally on developers' laptops and deploying them to production in cloud data warehouses, with features such as REST API ingestion, exploration, transformation, and deployment. With a rapidly growing community, the platform aims to provide a cohesive data engineering experience by offering a context layer that facilitates end-to-end workflows, while also accommodating the needs of small teams and scaling to enterprise levels with additional features and support.

The blog post explores the concept of ontology-driven schema evolution in data engineering, highlighting the challenges of managing evolving data schemas and the risk of exposing sensitive information like PII when new columns are added. It proposes a solution that involves creating an access policy described in plain English, converting it into a natural-language ontology, and using this ontology as a runtime policy to evaluate each column. The approach leverages both deterministic interpreters for clear cases and LLMs for ambiguous cases to decide which columns to include in an analytics view, ensuring only analytics-safe data is exposed. It demonstrates how ontology provides a consistent policy framework that adapts automatically to schema changes, separating the policy from code and allowing seamless updates without altering the underlying pipeline. The blog emphasizes the importance of maintaining an ontology for data safety, which remains valid even as schemas change, and suggests that the approach is particularly useful for handling complex data patterns that can't be resolved by simple pattern matching alone.

Agentic Data Engineering with dltHub offers a free, one-hour course designed for data and analytics engineers who regularly use tools like Claude and Cursor, aiming to create lasting, production-ready pipelines. The course outlines a comprehensive workflow involving metadata-driven processes that ensure seamless transitions between pipeline generation, validation, transformation, and monitoring stages. It emphasizes the importance of maintaining context across various steps in the data engineering lifecycle, facilitated by the AI Workbench, which integrates schema, contract, and runtime information. By focusing on structured processes rather than code issues, the course prepares engineers to handle complex data tasks efficiently, enabling them to generate, validate, and deploy data solutions swiftly while avoiding common pitfalls like documentation gaps and credential mishandling. This approach effectively reduces development time from sprints to single sessions and allows for scalable, reliable data infrastructure management.

AI agents are proving adept at writing data pipelines, yet challenges remain in ensuring isolation, auditability, and safe promotion to production environments. A recent demonstration by Elvis Kahoro and Ciro Greco showcased a data stack optimized for AI agents using tools like dlt, an open-source Python library, and Bauplan, a Python-native lakehouse. The demo highlighted the utilization of AI-oriented CLI commands and toolkits to streamline the creation and management of data workflows, using agents like Claude to construct pipelines efficiently. This approach emphasizes the automation of schema inference, normalization, and incremental loading while maintaining flexibility and control over endpoints and data transformations. The integration with Marimo enables validation of local data before production deployment, reducing the traditional timeline from weeks to minutes and minimizing the risks associated with prototyping. Bauplan complements this by allowing safe iteration and hypothesis testing in isolated branches, ensuring a seamless transition from data ingestion to production deployment.

Ontology engineering, a field that's been quietly debated for decades, is gaining renewed importance as artificial intelligence (AI) and language models (LLMs) replace human roles in decision-making processes. Unlike traditional human decision-makers who rely on implicit tribal knowledge, these AI agents require explicit, well-defined ontological frameworks to avoid "hallucinations" or errors due to lack of context. Ontology engineering involves creating precise definitions and relationships within a domain to enable AI to act autonomously and accurately. This resurgence is driven by the inability of LLMs to infer unstated information, the removal of humans from decision loops, and the need for scalable workflows without constant human intervention. Ontology engineering provides the semantic infrastructure necessary for AI agents to understand and act on data, moving the decision-making logic from humans to machines. This shift not only reduces ambiguity but also enhances the quality and reliability of AI-driven actions. The future of ontology engineering lies in integrating this comprehension layer with AI agents, allowing them to read, reason, decide, and act within a well-defined semantic framework.