September 2025 Summaries

The collaboration between Starburst and AWS offers a streamlined solution for transforming raw data stored in Amazon S3 into governed, accessible datasets suitable for analytics and AI workflows. Utilizing Starburst Galaxy's no-code data ingestion capabilities, organizations can efficiently convert raw files into structured tables within Apache Iceberg, a system known for its scalability and support for schema evolution, time travel, and ACID transactions. This integration eliminates the need for complex custom pipelines and allows for real-time data insights, as data is continuously ingested and optimized for immediate querying. The process supports seamless integration with tools like Amazon QuickSight for visualization, enabling a rapid transition from data storage to live analytics dashboards. By reducing operational overhead and enhancing data governance, Starburst and AWS facilitate faster insights and decision-making, ultimately modernizing data lakehouse architectures on the cloud.

Universal, fast, and secure data access is crucial for developing effective AI data products, as AI cannot operate on undiscovered data. Traditionally, data has been siloed within organizations, leading to delayed analytics and limited AI model performance. The text emphasizes designing data products that connect data across various environments, whether on-premises, in the cloud, or in hybrid settings, to enhance AI applications. Examples include a global retailer using federated data access for demand prediction and a financial services firm utilizing data products with comprehensive metadata and lineage for AI-driven risk models. Robust data governance is highlighted as essential, with AI amplifying the risks of poor governance, necessitating embedded access controls and compliance mechanisms. Open architectures are recommended to avoid vendor lock-in, ensuring data portability and adaptability to evolving technology landscapes. The text advocates for federated data access, packaging datasets as reusable products, and incorporating explainability into data products to support AI model transparency and trust. Automation and assisted workflows are suggested to manage data product life cycles efficiently, especially in hybrid and multi-cloud environments. Aligning data products with business objectives and leveraging AI for transformation, pattern recognition, and decision outputs are crucial for maximizing AI-driven business value.

The Starburst AI Agent, built on the principles of Agentic AI, is designed to accelerate data velocity and improve the delivery of AI solutions by enhancing how data products are documented, queried, and collaborated upon. Unlike traditional Generative AI, which relies on reactive and generalized approaches, Agentic AI is proactive, adaptable, and autonomous, allowing it to perform complex tasks with minimal human intervention. It employs specialized agents to break down complex problems, offering benefits such as increased accuracy, efficiency, scalability, and reusability. The Starburst AI Agent empowers data stakeholders by using natural language queries to facilitate data management, reduce the time to business value, and improve the documentation and discovery of data products. This enables better data-driven insights and innovation by providing tailored responses for different user personas, ranging from executives to data engineers. By leveraging the Starburst AI Agent, companies can overcome traditional data access barriers, enabling faster and more insightful data solutions.

The EU Data Act, complementing the GDPR, is pivotal in shaping the data landscape by ensuring data sovereignty, promoting data sharing, and enhancing governance while supporting AI initiatives. Unlike GDPR, which focuses on personal data protection, the EU Data Act emphasizes access to both personal and non-personal data to foster innovation and competition. It became applicable on September 12, 2025, requiring organizations to comply with regulations that include data residency, interoperability, and secure data sharing. Starburst assists organizations in navigating these regulations by offering technologies that ensure data sovereignty, facilitate data portability, and support interoperability without vendor lock-in. This includes capabilities like federated data access, multi-region deployment, and self-service data exploration, which help organizations adhere to the EU Data Act's requirements while enhancing their data architecture for broader analytics and AI applications.

The text provides a historical overview of the development and evolution of three popular open-source frameworks for data lake analytics: Apache Hive, Trino (formerly PrestoSQL), and Apache Spark. It highlights their journey towards achieving three key features: SQL support, performance, and durability. Initially, Apache Hadoop provided a foundation for Hive, which was created by Facebook in 2010 to offer a SQL abstraction layer over Hadoop's Java MapReduce API. Trino emerged in 2012, also at Facebook, as a solution for faster query execution by maintaining a separate compute cluster and offering federated queries across multiple data systems. Apache Spark was developed in 2014 at UC Berkeley’s AMPLab, focusing on performance by utilizing in-memory caching and resource allocation strategies. Over time, Spark added SQL support, and Hive introduced the LLAP framework for enhanced performance. In 2022, Trino incorporated fault-tolerant execution, aligning with the durability feature. Each framework ultimately achieved the three core attributes, making them essential tools in the realm of data analytics.

Starburst, known primarily as a SQL query engine, also serves as a versatile platform for ETL workloads, accommodating both SQL-oriented and Python-based data pipelines. Its evolution from Trino, originally developed for interactive querying, has seen widespread adoption for ETL tasks, replacing Hive in many contexts due to its superior query execution speed. Starburst's architecture employs a distributed processing mechanism using a directed acyclic graph (DAG), enhancing speed by not persisting intermediary data to disk and instead functioning like a streaming engine. However, initial limitations in handling long-running queries and memory-intensive tasks have been addressed by incorporating fault-tolerant execution (FTE), which allows for more reliable, stage-by-stage processing. Data engineers can construct pipelines using SQL or Python, with PyStarburst and Ibis offering Dataframe API alternatives for executing Python code in Trino clusters. The platform also supports orchestration tools like Airflow and Dagster, making it a robust choice for transformation processing jobs, further bolstered by its fault-tolerant execution mode and flexibility in using SQL or Python.

Starburst Galaxy, leveraging Apache Iceberg, offers a comprehensive data lakehouse architecture that enhances analytics and AI workloads by improving data accessibility, governance, and scalability. Recent updates to Starburst Galaxy include support for the latest Iceberg v3, which enhances performance and flexibility for large-scale analytics, and introduces innovative features like Iceberg branching for version control and support for AWS S3 file ingestion. These updates simplify data management by automating maintenance tasks and optimizing performance, thereby reducing operational burdens. Additionally, Starburst Galaxy supports AI workflows through features like AI-Powered Auto-Tagging and Starburst AI Workflows, which facilitate seamless integration of AI initiatives by making data instantly usable and aligning collaboration with governance. The platform's unification of ingestion, maintenance, and AI features underpins its claim as the most complete environment for Iceberg-based data lakehouses, providing a reliable foundation for both analytics and AI in the cloud.

Apache Iceberg's branching and versioning capabilities offer a robust framework for managing data changes in data lakehouses by allowing for safe experimentation and collaboration akin to Git branches in software development. These features enable data teams to isolate and test transformations, run large backfill jobs, and conduct what-if analyses without impacting production datasets. Branches in Iceberg are dynamic references that can evolve with new commits, offering flexibility in managing table changes over time. Unlike snapshots, which are immutable, branches allow for ongoing modifications, while tags serve as fixed pointers to specific snapshots. The branching functionality, available in platforms like Starburst Galaxy, enhances data management by simplifying workflows such as partition overwriting and providing a cleaner alternative to the traditional MERGE statement. While powerful, the current implementation of Iceberg branching has limitations, such as the lack of support for catalog-level branching and advanced retention policies. Nonetheless, Iceberg branching significantly enhances the safety, flexibility, and manageability of data lakehouses, making it a compelling choice for organizations using Starburst's query engine for Iceberg workloads.

Starburst has announced the release of its Enterprise 476-e LTS, emphasizing advancements in multi-cluster data architecture, enhanced AI capabilities, and improved performance metrics. This release includes the introduction of the Starburst Portal, which integrates the Starburst Data Catalog and Starburst Gateway to facilitate seamless cluster management and metadata control. The update also features support for Apache Iceberg v3, offering tools like branching for table version control, which enhances data governance and collaboration. The Starburst AI Agent is enhanced to enable natural language data interactions, allowing users to generate insights without needing deep SQL knowledge. Additionally, the release boasts significant performance improvements, with faster query processing and reduced CPU time, alongside an upgraded ODBC driver for better compatibility with BI tools. The release also brings several configuration changes and emphasizes the transition to AWS Glue v2, urging users to adapt to these new configurations to ensure future compatibility and performance.

Apache Iceberg v3 is an advanced open-source table format designed to enhance data governance, performance, and versatility in data lakes by introducing key features such as binary deletion vectors, richer data types, nanosecond-precision timestamps, and built-in row lineage. These enhancements allow for faster and more efficient data operations, particularly in high-throughput scenarios like change data capture (CDC), while also supporting complex data structures, such as semi-structured data and geospatial analytics. Iceberg v3's capabilities are supported by Starburst Enterprise and Starburst Galaxy, which provide improved query performance and expanded use cases for analytics on large datasets. Default values for new columns facilitate schema evolution, reducing the need for complex ETL processes. The row lineage feature strengthens data governance by tracking a row's history, ensuring compliance, and enabling robust auditing. This latest iteration of Iceberg positions it as a leading table format in the open data ecosystem, offering users powerful tools to manage and analyze data with greater efficiency and reliability.

Apache Iceberg's architecture utilizes snapshots for version control, allowing features like time-travel querying and rollback functionality without impacting query performance, as queries primarily interact with the current version. However, maintaining multiple snapshots increases the storage footprint on data lakes since each version's data files must be retained. While data lake providers allow extensive data storage, they charge for it, necessitating strategies such as expiring older snapshots to manage storage costs effectively. Iceberg's Merge-on-Read strategy, which handles updates and deletes without updating existing files, contributes to storage efficiency by minimizing the need for additional storage beyond what is currently used, although compaction processes can increase storage temporarily. Consequently, regular maintenance, including snapshot expiration and orphan file cleanup, is recommended to reclaim storage while balancing the need for historical version retention.