Query Federation Made Simple at Comcast

Comcast has developed an advanced hybrid analytics platform that efficiently integrates diverse data storage systems, enabling seamless data access for users across both on-prem and cloud environments. Initially faced with data silos spread across traditional RDBMS, data warehouses, NoSQL databases, and data lakes, Comcast's engineering team, led by Bryan Aller and Pavan Madhineni, sought a scalable solution to unify data access without relying on extensive ETL processes. By implementing a query engine using Trino and later integrating Starburst's enterprise-ready connectors, they created a Trino-to-Trino connector allowing for seamless communication between cloud and on-prem instances, effectively managing identity and security while abstracting data storage complexities. This innovation has enabled Comcast to leverage a data mesh approach, allowing analysts and data scientists to perform real-time analytics with standard SQL without duplicating data. The platform supports high-performance queries, significantly reducing processing times, and has been instrumental in handling large-scale data pulls of 250-300 TB daily, demonstrating both scalability and speed for hundreds of users accessing thousands of datasets.