September 2025 Summaries

The article from The Scalable Way by Dyvenia discusses challenges and solutions related to extracting data from SAP systems, particularly in light of the decommissioning of the pyRFC Python library. SAP's Remote Function Call (RFC) mechanism allows for system integration and data exchange, but handling large data volumes presents difficulties such as row size restrictions, performance issues, and data type conversions. The article highlights the development of a new Python library that aims to replace pyRFC, focusing on improved data ingestion and integration into modern Python workflows. It emphasizes the importance of efficient connection management, data filtering, and session handling to mitigate overhead and improve performance. Dominik, the tech lead of the project, shares insights on overcoming these challenges using a C++ connector, which offers better session stability, faster type conversions, and more efficient handling of large datasets. The discussion underscores the need for advanced connectors that integrate seamlessly into Python environments, facilitating the use of SAP data in analytics and machine learning applications.

MotherDuck has expanded its serverless data warehousing services, built on DuckDB, to Europe, ensuring operations and data remain within the EU, which is advantageous for companies seeking fast analytics without cross-Atlantic latency. This expansion is significant for European businesses like Trunkrs, which benefits from rapid query processing without the complexities of large systems. In conjunction with this, dlt has introduced native support for loading data into DuckLake, Ducklabs' open table format, through both managed and direct loading methods, allowing for flexible data management. Companies can either use MotherDuck’s engine for seamless data loading or opt for direct loading with dlt, which now supports multiple backends such as DuckDB, SQLite, and Postgres, with plans to add MotherDuck catalog support. The EU launch includes a waitlist as MotherDuck aims to be fully operational in the region by the end of October.

Adrian Brudaru, Co-Founder and CDO, explores the evolving landscape of AI strategies, emphasizing the concept of "scaled mediocrity" as a transformative force in enterprise applications. By leveraging AI to perform "good enough" tasks at scale, such as classifying millions of support tickets or generating draft content, businesses can achieve significant efficiency gains. The future of AI lies in hybrid, multimodal systems that integrate retrieval, generation, and tool use across various data types, enabling complex workflows previously deemed science fiction. Despite the hype around AI, lasting value is found in rethinking old problems and fostering innovation in fields like drug discovery and materials science. Brudaru criticizes "AI-washing" and advocates for rebuilding platforms from the ground up to achieve real innovation. The journey from incremental improvements to a 10x transformation at dltHub highlights the importance of creating LLM-native environments that fundamentally change data engineering processes, with a focus on robust data quality testing and scalable, integrated workflows.

Reshef Sharvit, a Principal Engineer at Skyhawk Security, explores the complex question of identifying the greatest UFC fighter of all time by using a full-stack data pipeline comprising dlt, dbt, and Metabase. He argues that determining the UFC GOAT requires more than just surface-level statistics like win-loss records, suggesting a need for deeper analysis using a set of 15 metrics. To achieve this, he utilizes data scraped from ufcstats.com and Wikipedia, which is then loaded into a PostgreSQL database using dlt for schema inference and incremental loading. dbt is employed to transform the raw data into analysis-ready views with SQL-first, version-controlled transformations and built-in testing for data integrity. Finally, Metabase is used to make the data accessible to non-technical users through intuitive visualizations. Sharvit finds this combination of tools effective and likens PostgreSQL's enduring utility to Jon Jones' lasting prominence in the UFC, despite controversies and challenges both have faced.

Microsoft Fabric offers a comprehensive suite for data engineering, data science, and business intelligence, but it lacks a built-in data quality (DQ) engine, leading to fragmented and inconsistent quality checks across services. dltHub addresses this gap by providing a code-first, Python-based framework that integrates seamlessly with Microsoft Fabric to manage data quality throughout the entire pipeline lifecycle. dltHub's solution includes source profiling, schema enforcement, pre-load validation, and controlled data loading, significantly reducing the operational burden on small teams by proactively preventing data issues before they cascade downstream. It also offers robust protection for personally identifiable information (PII) by detecting and masking sensitive data before storage, enhancing compliance and privacy. By acting as a gatekeeper, dltHub ensures that only validated, quality-checked data enters the lakehouse, thus increasing trust in analytics and reducing the need for constant firefighting over data quality issues. This integration simplifies end-to-end pipeline management and helps small teams deliver reliable, trustworthy data products without the complexity of managing multiple tools.

Aman Gupta's blog post outlines effective strategies for protecting Personally Identifiable Information (PII) using dlt and Pydantic, emphasizing the importance of enforcing privacy policies through automated data contracts rather than relying on static documentation. The article describes four architectural patterns—Sanitizer, Gatekeeper, Guardian, and Quarantine—that transform Pydantic models into active data ingestion gatekeepers, each addressing different aspects of PII management, from masking and blocking to monitoring evolving data schemas and redirecting exceptions. These frameworks ensure compliance with global data privacy regulations by integrating PII protection into data pipelines, enabling real-time governance and minimizing the risk of privacy breaches. By employing these methods, organizations can efficiently manage sensitive data, ensuring that only schema-compliant information is processed, thereby maintaining data integrity and regulatory compliance.

Adrian Brudaru's microblog discusses the limitations of using pandas' `df.to_sql()` for data loading in production environments and introduces a workshop at ODSC West 2025 to address these challenges. The blog highlights the need for transitioning from simple scripts to robust, automated pipelines due to issues like memory constraints, the need for incremental data loads, and resilience against schema changes. It suggests adopting a higher-level, declarative approach using tools like the open-source Python library dlt, which simplifies the management of streaming, state, and schema evolution. The workshop promises hands-on experience in building efficient data pipelines that can handle large files, process only new data from live APIs, and adapt to changes in data schema, thus enhancing data loading capabilities for production-ready environments.

Aman Gupta's blog post explores the complexities of managing Slowly Changing Dimensions Type 2 (SCD2) in nested data structures within data warehouses, focusing on the use of the dlt library to automate this process. SCD2 allows for the tracking of historical data by inserting new records instead of overwriting existing ones, and dlt simplifies this by managing SQL generation and versioning. The article demonstrates how dlt handles nested JSON records, generates SQL for maintaining historical changes, and evaluates the cost implications of different SCD2 strategies using BigQuery. Through practical examples and benchmarks, it highlights how incremental extraction is more cost-effective than non-incremental methods, and discusses the impact of varying nesting depths on query costs. The blog encourages readers to experiment with these concepts using an interactive Colab notebook and provides resources for further learning.