October 2023 Summaries

The blog post provides a comparative analysis of how major platforms like Starburst Galaxy, Databricks, and Snowflake handle data catalogs and table format catalogs, specifically focusing on Apache Iceberg and Delta Lake. Databricks has developed the Unity Catalog, supporting Delta Lake with governance features, while Snowflake has introduced a proprietary Iceberg catalog, currently in private preview, which aims to centralize Iceberg table metadata but has limited openness to external engines. Starburst Galaxy, leveraging the open-source Trino engine, promotes an open ecosystem through its Gravity data catalog, supporting a wide range of data sources and formats, including Delta Lake and Iceberg, without vendor lock-in. The discussion highlights the tension between vendor-controlled metadata ecosystems and more open architectures, underscoring the importance of metadata management in modern data infrastructure strategies.

In the logistics industry, which is lagging in digitization, 7Bridges collaborates with clients across various sectors to streamline their supply chains using an AI-powered management platform that addresses inefficiency and disruption. Simon Thelin, a lead data engineer at 7Bridges, highlights how Starburst Galaxy enhances the platform by consolidating scattered data and providing advanced simulation tools for optimizing shipments, exemplified by their work with Clinigen to improve medicine delivery logistics. Thelin's exploration of Trino led him to integrate Starburst Galaxy, which acts as a unifying framework for decentralized compute and centralized storage, enhancing data accessibility and enabling faster insights and development across the company. This integration not only improves operational efficiency but also fosters collaboration and innovation within the organization, ultimately benefiting clients by providing more accurate and actionable insights.

Starburst Enterprise Platform (SEP) 427-e STS, the successor to the 426-e STS release, is based on Trino 427 and introduces a range of new enterprise features and enhancements alongside the updates from the Trino 427 release.

Delta Lake, originally developed by Databricks and later open-sourced, offers features such as classical Data Manipulation Language (DML) statements and ACID-compliant transactions through a transaction log called DeltaLog, enabling time-travel querying by maintaining table versions. It integrates with Trino via the Delta Lake Connector, and Starburst Galaxy provides an accessible cloud-based platform to leverage this integration with Great Lakes connectivity, allowing users to utilize popular cloud object stores like Amazon S3. The blog post provides a practical demonstration of creating a schema and a table using Delta Lake, inserting data, and viewing table versions stored in S3, showcasing DeltaLog's role in tracking changes. The post emphasizes the ease of starting with Starburst Galaxy and encourages readers interested in technical details to explore further resources.

Starburst Galaxy has introduced Python DataFrames, currently in public preview, allowing users to utilize the DataFrame API, commonly associated with Spark, for data manipulation within the Starburst environment. The blog post by Lester Martin provides an overview of setting up the environment, including necessary installations like Python and pip, and details the process of using PyStarburst, a Python library for interacting with Starburst Galaxy. The post includes step-by-step examples demonstrating the creation and manipulation of DataFrames, showcasing operations such as table selection, filtering, joining, and sorting data from the TPCH dataset. It highlights the ease of chaining methods to perform data operations and suggests using the sql() method as an alternative for executing SQL queries directly. The post emphasizes the flexibility of PyStarburst for data engineers who prefer programming over SQL, while also noting that complex SQL queries can be simplified using the DataFrame API, which ultimately translates the operations into efficient SQL queries executed by the Trino engine on Starburst Galaxy.

Dell Technologies and Starburst have expanded their partnership to enhance AI capabilities through improved data collection and analytics, addressing the challenges of data silos in various industries. By integrating a modern data lakehouse offering, they provide a unified access layer for analytics tools, enabling faster insights and reducing inefficiencies for data engineers. This collaboration aims to offer organizations a single point of access to their data, regardless of its location, and minimizes the need for physical data movement, which can help manage costs and comply with regional data regulations. The joint solution is designed to accelerate innovation, modernize data estates, and drive effective analytics and AI outcomes, benefiting companies of all sizes across sectors like financial services, healthcare, manufacturing, and oil and gas.

Adam Ferrari, newly appointed SVP of Engineering at Starburst, shares his enthusiasm for joining the company to drive the data lake analytics revolution. He reflects on his extensive experience in data warehousing and analytics, emphasizing two key lessons: the importance of data discovery beyond curated data warehouses, and the necessity of massively parallel and scalable cloud architectures. Drawing from his time at Salsify, where he led the creation of a comprehensive data lake, Ferrari highlights the benefits of a data lake architecture, which allows for incremental progress, cost-effective experimentation, and broad data utilization across an organization. He praises Starburst's architecture for enabling a seamless integration of existing systems with new experimental data sets, fostering innovation without the need for wholesale infrastructure changes. Ferrari believes that Starburst's focused approach in the data lake space positions it to be a leader in the industry, and he is eager to contribute to its mission of making data lakes a central part of data platforms.

Combining AWS services with Apache Iceberg tables enables companies to create robust and cost-effective data lakes that enhance big data analytics capabilities. AWS offers a comprehensive suite for building enterprise-grade data architectures by leveraging open-source table formats like Hive, Iceberg, Delta Lake, and Hudi, which provide advanced features such as ACID transactions, schema evolution, and time travel queries. These formats, in conjunction with open compute engines like Hive, Trino, and Spark, facilitate high-performance analytics. AWS services such as Amazon Athena, Amazon EMR Trino, and AWS Glue support these frameworks by providing scalable, secure, and efficient data management and processing capabilities. Starburst Galaxy, based on the Trino query engine, further enhances this ecosystem by offering a unified analytics platform that democratizes data access while ensuring compliance and governance. Through a product-based approach, Starburst enables companies to build curated data products, thus improving data access and accelerating decision-making processes across AWS infrastructure.