November 2025 Summaries

A data product is a curated and accessible dataset with embedded metadata, designed for easy discovery and consistent use, primarily created by domain teams with specific business purposes in mind. Unlike raw data, data products emphasize ownership, governance, and accountability, ensuring high integrity and portability, thus addressing traditional collaboration challenges within organizations. Data products apply product thinking to data management, involving active lifecycle management and stakeholder feedback, similar to product management practices. They are particularly valuable for artificial intelligence (AI) initiatives, as they provide the context-rich metadata necessary for AI models to generate accurate results, reducing issues like AI hallucinations by offering clear data lineage, ownership, and quality controls. Platforms like Starburst facilitate the creation and management of data products at scale by enabling universal data access, integrating security and governance features, and enhancing AI and analytics capabilities with metadata-rich datasets. Data products serve as actively managed and governed assets, distinct from traditional data catalogs, making them crucial for organizations aiming to leverage AI effectively and maintain compliance across varied data sources.

The text discusses how investment firms seek to enhance their data velocity and decision-making processes to gain a competitive edge, known as "alpha," in financial markets. It highlights the challenges of data provisioning delays, restricted data access, and compliance hurdles, which slow down decision-making. Starburst offers a solution by providing secure, federated data access that allows firms to perform governed analytics and AI without centralizing data. The document outlines a comprehensive approach for financial services to adopt AI and analytics, including aligning outcomes, mapping data, implementing data governance, federating data access, publishing data products, enabling AI/ML workflows, and ensuring compliance and audit. This method aims to eliminate decision latency, enhance performance, and scale AI adoption effectively. The text also emphasizes the importance of a structured 30-60-90 day plan to achieve measurable success in AI projects, encouraging firms to pilot high-value use cases and expand their applications to gain data-driven alpha.

In the evolving landscape of financial services, speed is no longer the primary competitive edge; instead, data intelligence now defines the frontier. The focus has shifted from merely acting quickly to making informed decisions by effectively connecting, governing, and learning from data in real time. Legacy data architectures, which trap information in silos, are becoming obsolete as firms adopt adaptive, federated data ecosystems that prioritize insight over mere data movement. Starburst plays a pivotal role in this transformation by offering secure, federated access to data without relocation, ensuring consistent and auditable insights for faster decision-making and robust AI applications. This approach enables firms to engineer alpha by integrating distributed data into a cohesive, governed framework, facilitating the rapid discovery of market signals and enhancing client experiences. The emphasis is now on the effective utilization of data, with Starburst enabling precise trade analyses and real-time data access that empower traders and analysts to make smarter investment decisions.

Data lakehouses and data federation are converging to address significant data architecture challenges by enhancing data access and governance beyond the capabilities of traditional data warehouses or lakes. In modern enterprise environments where data is dispersed across multiple cloud and on-premises systems, centralized data storage is impractical due to cost, regulatory issues, and slower data pipelines. Data federation enables access and querying across diverse data sources without moving data, while data lakehouses provide the necessary storage, metadata, and governance foundation. Technologies like Apache Iceberg enhance transactional support and governance, making federated architectures viable at scale. By separating storage and compute, lakehouses allow for independent scaling of resources, optimizing performance, cost, and governance needs. Federated architectures offer a unified governance layer, enabling consistent policy enforcement across data. Modern data lakehouses also improve federated query performance through features like metadata caching and adaptive query planning. For AI workloads, federated architectures facilitate access to diverse training data while maintaining governance, supporting the concept of Lakeside AI, which integrates analytics and AI through federated access. The Starburst Icehouse Architecture exemplifies the integration of data federation and lakehouse principles, enabling federated queries without sacrificing performance or governance.

Starburst, built on Trino, enhances query performance through the Trino spooling protocol, which enables parallel retrieval of large result sets and improves performance with Starburst's ODBC driver. This protocol provides various client retrieval modes, such as STORAGE, COORDINATOR_STORAGE_REDIRECT, COORDINATOR_PROXY, and WORKER_PROXY, each tailored to different network, security, and performance needs. The spooling protocol allows data to be extracted from a Starburst cluster by persisting query outputs and enabling direct downloads, thus reducing the load on the Trino coordinator. The protocol supports multiple clients, such as Starburst ODBC driver V3 and Trino JDBC driver, and automatically defaults to traditional methods if a client doesn't support spooling. The configuration options aim to optimize network security and operational risk, offering scalability and flexibility in deployment, particularly in environments like Kubernetes, where certain modes may be less feasible.

Enterprises are increasingly focusing on operationalizing AI to deliver tangible business value, as AI transitions from experimental phases to production. Despite a high failure rate of AI projects in delivering measurable returns, mainly due to task-specific pilots that were hard to integrate, organizations are learning that success hinges on data foundation, governance, and workflow integration. Key areas where data-driven solutions are proving effective include fraud detection, supply chain optimization, and customer engagement, which require blending structured and unstructured data. As enterprises strive to make data AI-ready, they face challenges such as data discoverability, fragmentation, unstructured data, governance, and tool sprawl, which can be addressed through AI-assisted documentation, federated queries, vector searches, and robust governance models. Privacy and governance remain crucial, demanding that AI governance extends beyond data tables to include models and prompts, with strategies like minimizing data movement and leveraging open standards. Recent breakthroughs in data platforms, such as AI SQL functions, vector search, model governance, and agent-assisted discovery, are shifting the focus from experimentation to scalable, governed AI implementation. The future success of AI in enterprises will depend on unifying data, consistent governance, and designing for real-world action, with platforms like Starburst addressing these challenges through federated queries and Iceberg-based data access.

Moacyr Passador, a Pre-Sales Readiness Manager at Starburst, details his experience using Starburst's own AI tools and data platform, Starlake, to address the company's business needs, demonstrating the application of Starburst on Starburst. By integrating various data sources like Homerun Presales and Salesforce into Iceberg tables in a data lake, Passador's team leveraged AI functions to explore and optimize data processing, enabling them to generate insights rapidly. The project involved solving data access challenges, formulating specific business questions, and utilizing AI to analyze pre-sales activities and assess hiring needs. Furthermore, Passador highlights the iterative process of creating data products, including metadata generation and query optimization, to enhance decision-making and performance while using Starburst's AI agent for operational scalability. This internal initiative not only improved the efficiency and confidence of Starburst's Go-to-Market team but also exemplified the potential for self-service data experiences, emphasizing the advantages of using their platform for internal applications.