How Starburst’s data engineering team builds resilient telemetry data pipelines

Isaac Obezo, a staff data engineer at Starburst, discusses the implementation of Starburst Galaxy and dbt in building resilient telemetry data pipelines. He contrasts this modern approach with traditional data warehousing, highlighting challenges such as pipeline fragility and maintenance complexity. Starburst Galaxy, a fully managed data lake analytics platform, serves as a non-destructive, abstraction layer across various data sources, facilitating a more flexible and efficient data organization process. By using dbt alongside Galaxy, Starburst's team can build, test, and document models more effectively, enhancing data quality and enabling a domain-oriented workflow. This combination allows for increased scalability, agility, and autonomy, as it reduces the dependency on multiple systems and focuses on a single development lifecycle, making it easier to iterate and improve analytical models. Through this integration, the team shifts from maintaining systems to building new projects, fostering a data-driven culture that can adapt to change without significant disruption.