May 2024 Summaries

Data federation, exemplified by tools like GlareDB, simplifies the process of querying and joining data from multiple sources without the need for complex infrastructure or data migrations. In a scenario where New York City real estate sales data is stored in Snowflake and a lookup table for borough names is kept in PostgreSQL, GlareDB allows users to join these disparate data sets efficiently. By using specific functions such as read_postgres() and read_snowflake(), data from both databases can be queried and combined into a unified pandas DataFrame, thus bypassing traditional extract, transform, load (ETL) processes. The demonstration, which can be followed in a Jupyter Notebook or via video walkthroughs, illustrates how GlareDB facilitates data federation by treating results from different databases as though they were in the same location, enabling seamless querying and data integration.

Tal Gluck's blog post guides readers on setting up a dbt pipeline that integrates federated data sources using GlareDB Cloud, allowing users to treat data from multiple sources as one without additional data extraction or infrastructure setup. The post explains how to use GlareDB to simplify the process of joining data from various sources, such as Snowflake, BigQuery, Postgres, and files hosted on platforms like GitHub, by leveraging dbt's capabilities and the Postgres adapter. It highlights the ease of creating dbt models and incorporating data from different origins without requiring complex data movement operations, emphasizing GlareDB's ability to function as a unified SQL-accessible platform. The author shares personal experiences and provides practical steps to demonstrate the seamless integration and transformation of data using dbt and GlareDB, while also showcasing advanced functionalities like pre-hooks, file uploads, and data exports to object storage.