Company
Date Published
Author
Adam Stone
Word count
3293
Language
English
Hacker News points
None

Summary

Netlify's data team successfully migrated their data stores from Databricks to Snowflake, a complex effort involving over 500 dbt models, dozens of Mode reports, and re-engineering of Airflow data ingestion pipelines. To manage the migration, they broke it down into bite-sized chunks, prioritized tasks based on dependencies, and maintained open communication throughout the process. The team utilized checklists to track progress, relied on each other for reviewing PRs and resolving data discrepancies, and shared knowledge to overcome challenges in SQL functions and analytics logic between Databricks and Snowflake. The migration required adapting to differences in SQL dialects, data types, and timestamp handling, but ultimately resulted in a smoother transition with improved performance and reliability.