Refresh your data faster using incremental models

In the third part of the lakehouse ETL series with dbt and Trino, the article explores the use of incremental models to enhance data refresh speeds by limiting the amount of data loaded into target tables, thereby reducing both loading time and compute costs. Incremental models are initiated by transforming all source data during the first run and subsequently filtering only specific rows for updates, using configurations such as `materialized='incremental'` and unique keys to prevent duplicates. The article discusses several incremental strategies supported by the dbt-trino adapter, including append, delete+insert, and merge, which vary in effectiveness based on data volume and unique key reliability. Key examples include using the append strategy for a clicks dataset and the merge strategy for sessionized clicks, ensuring efficient data handling with dbt's SQL-based transformations. The article also highlights Starburst's role in enhancing Trino features and encourages community engagement through Slack channels for further learning and exploration.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.