September 2024 Summaries

Trino is a distributed SQL query engine designed for running analytics on large datasets across various data sources, such as data lakes and lakehouses, without storing the data itself. Originally developed by Facebook as Presto in 2012 to expedite queries on massive datasets, Trino was forked in 2019 by its co-founders to enhance its features and adoption in the data community. Trino's architecture supports massively parallel processing and a connector-based system, allowing it to efficiently handle federated queries across multiple data systems. It is particularly suited for analytics workloads, offering ANSI SQL compatibility, which facilitates integration with other data analytics systems. While Trino is open-source, Starburst, the main contributor to its development, provides managed versions like Starburst Galaxy, adding proprietary features and simplifying deployment for organizations lacking the resources to manage Trino independently. Compared to other tools like Spark and Presto, Trino excels in analytics performance but is not directly in competition with Spark, which is better suited for ETL workloads.

Emily Sunaryo, a Developer Relations Engineer, explores the use of Starburst Galaxy and Trino in building data applications, demonstrating their potential to enhance query performance and expand their utility from data engineers to app developers. The project utilizes open-source NYC rideshare and weather data to create a rideshare analytics dashboard, analyzing factors like location, weather, and time on traffic data points. By using the trino-js client, developers can leverage Trino’s powerful SQL engine to manage backend data requests without directly handling the complexities of Trino's REST API. The process involves integrating Starburst Galaxy with modern front-end technologies, configuring catalogs in AWS S3, and utilizing schema discovery for data management. This initiative highlights the practical application of big data in creating independent, custom data applications that derive analytical insights for business decisions, distinguishing them from traditional BI tools. The article concludes with a promise of further exploration into implementing the trino-js client within a React application in a subsequent blog post.

In the rapidly evolving world of data lakehouses, Apache Iceberg and Delta Lake have emerged as two leading table formats, each offering unique advantages and converging features that blur their distinctions. Originally developed by Netflix and Databricks, respectively, both formats prioritize efficient data management capabilities such as ACID compliance, schema evolution, and time travel queries. Although once distinct, the competition between Iceberg and Delta Lake has driven convergence, leading to similar feature sets. However, their underlying mechanisms for metadata management differ; Iceberg uses manifest files while Delta Lake utilizes a Delta Log. The choice between these technologies often depends on the existing data ecosystem and specific organizational needs, with Iceberg being favored for its openness and compatibility with various compute engines, while Delta Lake offers deep integration with Databricks and Spark. As the industry moves towards a more unified data stack, Apache Iceberg's long-standing commitment to openness and its compatibility with Trino and other engines make it a compelling choice for organizations seeking flexibility and scalability in their data architectures.