July 2025 Summaries

Starburst has introduced a new feature that simplifies file ingestion from AWS S3 into data lakehouses, enhancing the capabilities of its data platform. This feature allows seamless, no-code ingestion of AWS S3 data, automatically converting raw data into structured Apache Iceberg tables. This addresses the complexity and inefficiencies associated with traditional batch ingestion methods, which often rely on multiple tools and manual processes. By integrating file ingestion into a governed platform, Starburst ensures secure, compliant use of data while reducing the complexity of data pipelines. The approach supports real-time analytics, data applications, and AI workflows, enabling organizations to quickly and efficiently build and maintain scalable, reliable data lakehouses.

Open data architecture is becoming essential for organizations leveraging Generative AI (GenAI), as it allows for interoperability, standard formats, and the separation of compute and storage layers, which are crucial for context-rich, responsive, and secure AI applications. Traditional data systems often create silos and vendor lock-in, hindering AI's ability to access comprehensive datasets across multiple domains, which can lead to underpowered models and compliance challenges. An open architecture enables data to be treated as a strategic asset, facilitating real-time data access, eliminating silos, and reducing the risk of regulatory infringements while allowing for the integration of new tools and technologies without extensive re-platforming. This architecture supports the creation of collaborative data products, which are essential for business intelligence and AI initiatives, by allowing teams to query data across distributed sources seamlessly. The Icehouse model of data architecture, using technologies like Apache Iceberg and Trino, exemplifies this transition by offering a flexible and future-proof platform that can adapt to evolving technological and regulatory landscapes, ultimately accelerating AI adoption and innovation while reducing costs and technical debt.

In the realm of financial services, Banco Inter in Brazil serves as a case study in using data governance to drive organizational change and accountability. The bank implemented a data mesh architecture to democratize data access while introducing a chargeback model that links data consumption to accountability, enabling departments to assess and manage their data usage effectively. This initiative not only reduced costs but also improved transparency and fostered collaboration across operational silos. By aligning data usage with business outcomes and employing strategic data design, Banco Inter has enhanced its readiness for future projects and positioned itself for long-term success. The bank's approach underscores the importance of visibility, governance, and interoperability in achieving analytics maturity, offering valuable lessons for financial institutions aiming to modernize operations and leverage data as a strategic asset.

Trino is a powerful data processing and query execution engine that excels at accessing and integrating data across various storage systems due to its connector-based architecture. Unlike traditional database management systems that handle both storage and processing, Trino focuses solely on processing and querying, which can lead to performance bottlenecks, especially when using generic connectors like JDBC. These bottlenecks are mainly due to the slow, single-threaded nature of JDBC data transfer, which can severely limit throughput despite Trino's high processing capabilities. To mitigate these issues, Trino attempts to push down query operators to reduce the data transferred, but this is not always sufficient. System-specific connectors can sometimes alleviate bottlenecks through partition-aware extraction and increased processing pushdown, offering significant performance improvements over generic connectors. In cases where performance issues persist, best practices include exploring other Trino distributions with better connectors or storing periodic data snapshots in open formats in data lakes, where Trino performs more efficiently.

On-premises AI offers a viable solution for companies in regulated industries that require strict data governance and compliance, such as finance, healthcare, and the public sector, by allowing AI workloads to run within the company's infrastructure rather than relying on cloud-based solutions. This approach provides increased data security, control over infrastructure, and the ability to quickly implement AI projects without the need for extensive data migration. On-premises AI leverages technologies like Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) to enhance data accuracy and relevance while maintaining data sovereignty and regulatory compliance. Starburst's hybrid architecture, including the use of Apache Iceberg and Trino, supports these AI initiatives by enabling fast access to data across multiple sources, thereby enhancing governance and performance. This strategy is particularly beneficial for organizations that need to maintain data within specific geographic locations due to data sovereignty laws, enabling them to adopt AI more efficiently and securely.

The Starburst + Databricks AI Summit 2025 highlighted the collaboration between Starburst and Databricks, focusing on data access and compute interoperability. The event featured a variety of activities, including a speakeasy soirée attended by industry leaders and a demo showcasing Starburst Galaxy's integration with Unity Catalog, enabling governed, cross-engine interoperability. Starburst emphasizes providing choice in data access, allowing users to access data stored in various formats like Iceberg, Delta Lake, Hudi, or Hive, with the flexibility to run workloads regardless of where metadata is stored. This flexibility, coupled with interoperability with catalog options such as Hive Metastore and AWS Glue, aims to enhance the customer experience by minimizing metadata duplication and cutting storage costs. Starburst's commitment to choice-driven data management is underlined by its support for interoperable compute, allowing users to select their preferred catalogs and compute engines, ultimately providing a more flexible and efficient data management solution.