November 2023 Summaries

The author created a custom dlt integration to connect SAP HANA with Snowflake, allowing for the transfer of data between these two systems. This was achieved by leveraging dlt's SQL source feature and installing additional packages to handle SAP-specific requirements. The integration enables users to load data from SAP HANA into Snowflake, which is cloud-agnostic and can be run in various environments. The author highlights that this connector can also be used with other tools like dbt and airflow, making it a valuable addition to an SAP workflow. Further development plans include creating a verified source for SAP HANA and exploring the possibility of connecting SAP S/4 HANA with Snowflake.

Data lineage is vital for data engineers as it traces the journey of data from its origin to its destination, aiding in troubleshooting, regulatory compliance, and understanding the impact of upstream changes on downstream data. The text describes a demo project that utilizes dlt and dbt to establish data lineage, focusing on a skate shop's sales data from Shopify and physical stores, which is then loaded into BigQuery. The demo illustrates creating table, row, and column lineage using dlt's load_info feature, which captures schema changes during data ingestion. The process includes using dbt to transform raw data into a fact_sales table for analytical purposes, with lineage details visualized through a dashboard in Metabase. This approach enables tracking of data changes and lineage at multiple levels, providing valuable insights for maintaining robust data pipelines.

Cloud Functions are used to receive and process webhook requests from external sources such as GitHub, Slack, and HubSpot, which can trigger various actions like sending real-time alerts or storing data for later use. Cloud Functions provide a cost-effective way to handle low volumes of events with pricing unbeatable for serverless services. The setup involves creating a cloud function with an HTTP trigger, configuring the webhook URL in the external source's settings, and integrating the application with the desired platform. For example, setting up a GitHub webhook sends relevant data from pull requests or commits to BigQuery, while a Slack webhook processes messages sent in a channel and stores them in BigQuery, and a Hubspot webhook triggers specific conditions within an automation workflow and sends data to BigQuery.

Solving data ingestion for Python developers is a complex challenge that has seen various attempts at solutions, including SDKs and custom pipelines. The current state of data ingestion is characterized by trade-offs between simplicity and flexibility, with existing tools often lacking the perfect balance. A new approach, dlt, aims to bridge this gap by offering an approachable yet powerful solution designed specifically for Python users. This tool enables automation of complex tasks, concise code, and fine-grained control over data pipelines while fostering a community aspect that allows builders to share their content and insights.

dlt is an open-source Python library designed to streamline the process of loading unstructured data into structured datasets by providing automated schema inference and evolution. It supports scalable data pipeline building, allowing for deployments on micro workers or highly parallel setups, and offers features such as state management for incremental data extraction. This functionality is demonstrated through a project that ingests GitHub issue data into BigQuery using dlt, with orchestration handled by Dagster. Dagster enables the transformation of data pipelines into assets and resources, enhancing the robustness of the pipeline through features such as schema evolution monitoring and the use of configurable resources. The project also showcases how to orchestrate MongoDB verified sources using Dagster, employing the @multi_asset feature to separate data loading for each collection, which improves debugging and independence. The integration of dlt and Dagster allows for the rapid development and testing of data pipelines before production deployment.