September 2026 Summaries

Data federation has evolved from a data-virtualization tool for ad hoc analytics into production infrastructure that provides governed, unified access to distributed data without requiring full centralization. The shift is driven largely by AI agents, which require current context from operational databases, SaaS applications, warehouses, and data lakes, while copying every source through ETL pipelines can introduce delay, cost, and maintenance burdens. Improvements in query pushdown, connector coverage, semantic layers, caching, and support for standards such as the Model Context Protocol have addressed earlier performance and integration limitations. Data-sovereignty rules, including GDPR-related restrictions and the EU’s DORA requirements, also encourage organizations to analyze sensitive information where it resides rather than move it across jurisdictions. Federation is presented as complementary to data lakehouses: lakehouses manage centralized data with durable storage and transactional governance, while federation extends access to systems that remain distributed. Organizations evaluating federation platforms should assess relevant connector and table-format support, query pushdown performance, consistent row- and column-level governance, and whether AI agents receive the same governed access as business intelligence tools.

NetApp and Starburst position their partnership as a response to a common obstacle in enterprise AI projects: fragmented data spread across storage systems, cloud platforms, lakehouses, and legacy environments. NetApp’s AI Data Engine provides storage and contextual access to unstructured data, while Starburst supplies a federated SQL query layer that can access sources including ONTAP, StorageGRID, Snowflake, Databricks, Iceberg, Delta Lake, Hive, Hadoop, and Teradata without first creating additional data copies. The integration is software-based rather than a joint hardware appliance, with Starburst’s AIDA assistant helping plan queries, identify relevant sources, and cite answer origins. The companies argue that querying data in place can reduce preparation costs and tool complexity, while Stargate aims to place compute near distributed data in hybrid and multi-cloud settings. Governance is presented as a central feature through unified access controls, masking, audit trails, encryption, and compliance capabilities intended to support regulated industries.

On-premises data lake architectures remain relevant for organizations with data sovereignty, regulatory, latency, security, or long-term cost-control requirements that make moving large volumes of sensitive data to public clouds impractical. Modern implementations replace rigid Hadoop-centric designs with local object storage or HDFS, open table formats such as Apache Iceberg, Delta Lake, or Hudi, distributed SQL engines such as Trino, and governance layers for metadata, access controls, and lineage. These components separate storage from compute, support ACID transactions and schema evolution, and enable federated querying across on-premises, cloud, SaaS, and edge environments without requiring complete data centralization. Common deployment patterns include single-site lakehouses, multi-site federated clusters, hybrid cloud integrations, and edge-to-core systems for manufacturing and IoT data. While on-premises platforms offer greater control and low latency, they require capacity planning, hardware maintenance, disaster-recovery design, and specialized expertise; modernization commonly involves incrementally migrating Hive-based lakes to Iceberg-based lakehouses. Starburst presents its Trino-based platform as one option for federated access and governance, citing Lockheed Martin and GEODIS as examples of organizations reporting operational and query-performance improvements.

Modern data governance must address distributed data across cloud storage, warehouses, operational databases, and AI workloads without relying on costly centralization or data migration. The recommended approach places governance on the federated query access path, where identity authentication, role-based access control, attribute-based policies, dynamic row filtering, column masking, and centralized audit logging can be applied consistently at query time. Data products serve as reusable governed assets that combine certified business definitions, ownership, security rules, and lineage, helping human users and AI agents access trusted data under the same controls. Organizations can implement this model incrementally by governing one clearly owned business domain, classifying and tagging sensitive fields, attaching policies, publishing a certified data product, validating access through audit logs, and then expanding the pattern to other domains.

Starburst Enterprise 482-e LTS introduces incremental, append-only refreshes for Iceberg materialized views in public preview, enabling scheduled updates that process only new data and reduce recomputation costs, alongside Iceberg REST Catalog support in Starburst Metastore. The release reports 10% faster query performance in internal benchmarks and adds public-preview capabilities for queries that exceed available RAM through disk or object-storage backing, plus Apache Arrow-based PyStarburst result retrieval that can improve Python output performance. AIDA enhancements include token-usage monitoring, configurable in-chat tools, granular Skills access controls, expanded OAuth support for external MCP servers, and an updated data product enrichment agent that uses data sampling and business rules to produce metadata. Connectivity additions include general availability for Great Lakes and OpenSearch connectors and public previews of OpenAPI and kdb+ connectors. The update also includes breaking changes, such as removal of the packaged Hive Metastore Service, legacy Hadoop-based cloud storage support, the Snowflake JDBC connector, Alluxio support, and catalog-level file-system caching, requiring affected users to migrate configurations and services before upgrading.

Banks are deploying AI at scale across document and research copilots, fraud and anti-money-laundering monitoring, wealth-management support, and agentic systems that combine structured and unstructured data. Early gains have come from tools that summarize filings, transcribe meetings, search research, and automate routine employee tasks, while more complex applications use real-time transaction data to detect suspicious activity and support risk analysis. The central challenge is fragmented structured data, such as account histories and transaction records, which often resides in separate systems with different formats and access rules. The article argues that banks with the broadest production adoption prioritize unified, governed data access before selecting AI models, allowing systems to query data in place rather than requiring full replatforming. It cites reported examples of large employee adoption, faster data queries, and reduced financial losses, while emphasizing that data sovereignty, access controls, lineage, and auditability must be built into AI deployments to meet regulatory requirements and accelerate movement from pilot projects to production.

Context engineering is presented as a way for organizations using Amazon Redshift to make AI agents more reliable by supplying business definitions, metadata, governance rules, and vetted data products rather than only raw tables and columns. While Redshift supports structured analytics and low-latency reporting, its approximate 50-query concurrency limit, single-database catalog scope, proprietary storage, and limited visibility into data held elsewhere can constrain agents that need to explore multiple systems. The piece argues that a governed context layer can define metrics such as active revenue and churn, enforce row- and column-level access policies, and reduce agents’ need to guess which sources are authoritative. It recommends federating Redshift with data lakes, other warehouses, and SaaS applications instead of copying all data into another silo, and using the Model Context Protocol to give multiple agents and tools consistent access to the same governed definitions. Organizations are advised to account for Redshift connection, TLS, catalog, and maintenance considerations, inventory needed business concepts and sources, centralize policies, and validate vendor performance and cost claims against their own workloads.

Starburst Enterprise Platform 481-e STS introduces server-side scan planning for Iceberg through the REST Catalog API, allowing compatible catalogs to return exact file scan tasks and scoped credentials without requiring Starburst to process Iceberg metadata files. Its initial integration with Databricks Unity Catalog uses the Iceberg 1.7.0 `/plan` endpoint and Unity Catalog’s ScanAPI to centrally enforce fine-grained governance policies, including attribute-based access controls, row filters, column masks, and governed tags, for managed Iceberg tables queried through Starburst. Unity Catalog evaluates user policies, creates sanitized temporary data files, and provides Starburst only with authorized files and short-lived credentials, enabling consistent policy enforcement across Databricks and external engines without embedding Databricks-specific policy logic in Starburst. Starburst plans to support any Iceberg REST catalog implementing the endpoint, positioning the capability as part of broader industry efforts toward interoperable, centralized data governance.

Data masking protects sensitive information by replacing, obscuring, tokenizing, or encrypting values while retaining enough structure and relationships for analytics, with dynamic masking applying rules at query time and static masking creating permanently protected copies. It has become increasingly important as privacy regulations such as GDPR, HIPAA, and PCI DSS, along with wider organizational access to data, require organizations to balance security with analytical usability across finance, healthcare, retail, and other sectors. Native capabilities in platforms such as Snowflake, BigQuery, and SQL Server can protect data without changing many user workflows, but implementation remains difficult because masking may reduce query performance, complicate caching and optimization, fail when shared service accounts obscure end-user identities, fragment policies across platforms, and disrupt pipelines that rely on unmasked keys. Effective programs begin with high-risk use cases, establish reliable identity propagation, test performance impacts, maintain audit records, and use scalable classification or tag-based policies; the source presents Starburst as a platform that can support unified governance, identity pass-through, and cross-platform masking management.

Effective AI agents require a well-maintained context layer that captures not only accessible data but also the undocumented institutional knowledge, workflows, definitions, decision rules, and relationships among systems that employees use intuitively. The piece argues that agents can fail or make harmful decisions when they lack specific process context, such as approval requirements, customer-priority rules, legal retention obligations, or distinctions between similarly named business entities across databases. It groups business context into technical metadata, business definitions and metrics, and process knowledge, noting that the latter two often require substantial human documentation efforts because they may exist only in employees’ experience. Citing a survey in which only 7% of enterprises considered their data fully ready for autonomous agents, it contends that improving data access alone is insufficient when critical information has never been recorded as usable data. Organizations are therefore encouraged to make context engineering an ongoing practice, documenting evolving workflows, the rationale behind metrics and classifications, and cross-source data connections so agents can act more reliably within appropriate guardrails.

Query pushdown is a query optimization method that executes filters, aggregations, projections, and sometimes joins within the systems where data resides rather than transferring full datasets to a central query engine, reducing network movement, compute use, costs, and query latency. It supports both federated analytics across sources such as warehouses, databases, SaaS platforms, and lakehouses, and ingestion workflows that materialize selected results into tables or object storage for downstream analytics and AI uses. Its effectiveness varies by connector and source capabilities, SQL dialects, API limits, governance requirements, cross-region egress fees, and cloud pricing, making performance and cost outcomes difficult to predict. The Starburst team recommends verifying pushdown behavior with EXPLAIN plans, incorporating checks into development pipelines, using connector-specific settings and cost controls, materializing reusable federated results, applying dynamic filtering and fault tolerance where appropriate, monitoring source capacity, and enforcing row- and column-level governance before data is moved.

A context layer is described as a governed architectural tier that sits between enterprise data sources and the human or AI tools that use them, supplying verified business definitions, relationships, policies, and auditability rather than duplicating underlying data. It combines structure, meaning, and governance by cataloging data assets and valid joins, resolving identities and business rules across systems, and enforcing access controls, certification, and traceability at query time. Unlike a semantic layer, which primarily standardizes metrics and dimensions within analytics environments, a context layer also addresses cross-system entity matching, non-metric rules, steward approval, and context retrieval tailored to individual AI requests. The proposed workflow collects metadata from catalogs, pipelines, BI tools, and query logs; has people certify it; packages only relevant information for a question; and supplies that context before an AI agent reasons, reducing reliance on raw schemas and potentially unreliable inferences. The piece argues that organizations can adopt this model incrementally through governed data products and federated access to existing data sources, with Starburst presented as one platform supporting this approach.

SQL on anything describes a data federation approach that uses standard SQL to query and join data across disparate systems, including warehouses, lakes, operational databases, streaming platforms, and SaaS applications, without first copying everything into a central repository. Powered by compute engines such as Trino and enabled through source-specific connectors, federation can reduce delays associated with building data pipelines, preserve flexibility across hybrid, acquired, and regulated data environments, and provide broader context for analytics and AI applications. The approach does not eliminate the need for centralization, which remains useful for stable governed datasets, high-performance dashboards, model training data, and long-term historical records, but makes it a workload-specific choice rather than a default requirement. Effective federated access also depends on centralized governance measures such as row-level filters, column masking, role- or attribute-based access controls, semantic definitions, and unified audit trails, helping reduce risks introduced by duplicated data copies. Starburst presents Galaxy for managed cloud deployments and Enterprise for self-managed private, hybrid, or on-premises environments, both offering federated SQL through Trino.

As AI agents increasingly perform complex enterprise data analysis, their effectiveness depends heavily on access to complete, high-quality data and on the architecture used to retrieve and process it. Sending raw database extracts directly to an agent is presented as impractical for most large workloads because of high token costs, data-transfer overhead, and weaker processing efficiency, while granting agents direct database access allows database engines to execute optimized queries locally. A data virtualization or federation layer offers an alternative by giving agents a single interface to multiple data sources, though it introduces setup, software, and operational complexity. Its principal benefits include efficient joins across separate systems, centralized data discovery and access management, query-result caching for repetitive agent workloads, and specialized connectors that can better exploit source-system capabilities. The analysis concludes that tightly integrated native agents may be preferable where available, but virtualization can improve the quality, speed, and cost-effectiveness of data access for externally selected agents, particularly when enterprise data is distributed across many platforms.

Data lineage maps the flow, transformation, and dependencies of data from source systems through processing stages to dashboards, reports, models, and other consumption points, helping organizations trace unexpected results, assess change impacts, and meet compliance requirements. It is particularly important for regulated reporting, data quality investigations, machine learning governance, and operational resilience, but comprehensive implementation is difficult across heterogeneous platforms because of inconsistent metadata, schema changes, column-level transformation complexity, identity mapping issues, retention limits, and incomplete query-log reconstruction. The recommended approach is to begin with high-value use cases and table-level lineage, adopt interoperable standards such as OpenLineage, collect raw lineage events for iterative modeling in a lakehouse, and expand to granular column-level coverage where risks are greatest. Starburst is presented as a platform that can emit OpenLineage events, stream query activity through Kafka, federate lineage data with operational metadata, enforce access controls, and support near-real-time analysis, while emphasizing that lineage freshness, performance, governance, and careful temporal modeling are essential to maintaining reliable results.

Column-level security restricts visibility of individual database fields according to a user’s role, permissions, or attributes, allowing organizations to mask, hash, or hide sensitive data such as social security numbers, account numbers, and patient identifiers without changing the underlying datasets. It supports self-service analytics, AI and machine learning, and regulatory compliance with frameworks including GDPR, HIPAA, PCI DSS, and NIST by applying least-privilege and data-minimization principles across different users and workloads. Implementation can be complicated in multi-platform and federated environments because systems such as Snowflake, BigQuery, and Databricks use different policy models, identity context may be lost when pipelines run through broadly privileged service accounts, schema changes can leave new columns unprotected, and policies may affect performance, caching, external tables, materialized views, and derived datasets. Starburst positions column-level controls within broader governance that also includes role-based, row-level, and attribute-based access controls, enabling either source-system enforcement through identity or OAuth token passthrough or consistent engine-level masks and filters. Recommended adoption practices include beginning with a focused, high-value use case, defining precise masking behavior, planning performance optimization, and establishing governance, approval, schema-deployment, and testing processes before expanding protections across the data platform.

Starburst CEO Justin Borgman argues that enterprise AI depends on broad, timely access to organizational context, which is dispersed across databases, SaaS platforms, data warehouses, lakehouses, and institutional knowledge rather than residing in one location. He contrasts the traditional approach of centralizing data through repeated copying and migration, which he characterizes as costly, slow, restrictive, and prone to vendor lock-in, with data federation, which allows organizations to query data where it remains. The piece contends that recent improvements in federation performance make it a viable alternative to centralization and especially important for AI systems that require current, governed business context to operate effectively. It presents Starburst as a context layer for AI, emphasizing four requirements: federated data access, semantic alignment through curated data products, ongoing adaptation as data and business meaning change, and an Iceberg-centered lakehouse strategy for new data while retaining access to legacy sources.

Apache Iceberg tags in Starburst provide immutable named references to specific table snapshots, functioning like Git tags while Iceberg branches remain mutable pointers for ongoing work. Tags support audit-ready regulatory snapshots, reproducible analytics and machine-learning datasets, release checkpoints, and protection of important snapshots from expiration through configurable retention policies. Starburst uses extended CREATE BRANCH syntax to create, replace, query, and drop tags, including tags for current or historical snapshots and tags captured from a branch. Tagged data can be queried with standard time-travel syntax and retains the schema that existed at the time of the snapshot, supporting accurate historical records. The article also describes tiered retention strategies using short-lived weekly, medium-term monthly, and long-term annual tags to balance storage costs with historical-access and compliance needs.

AI initiatives often fail not because of inadequate models but because organizations lack accessible business context, with surveys indicating that few enterprises consider their data fully AI-ready and many struggle with data searchability and reuse. The discussion distinguishes between “locked library” knowledge in data silos, which can be made discoverable through connectors, cleaning, federation, and centralized catalogs, and the more difficult “unwritten book” of institutional knowledge that resides in employees’ experience and undocumented practices. Large language models can interpret information that is available to them but cannot reliably resolve conflicting definitions or create missing business meaning without human guidance, increasing the risk of arbitrary or inaccurate outputs. Context engineering therefore involves identifying gaps exposed by AI systems, eliciting knowledge from subject-matter experts, documenting it, and incorporating it into a semantic layer that provides consistent business definitions between raw data and AI applications. Combining federated access to data where it resides with ongoing human processes for capturing and maintaining institutional knowledge can help organizations build more reliable and trustworthy AI agents.

A query engine is software that plans, optimizes, and executes SQL queries against data stored in external systems rather than storing data itself, separating compute from storage to support data federation across data lakes, warehouses, and operational databases. Using source-specific connectors and metadata from metastores, it creates optimized query plans that employ techniques such as predicate pushdown, cost-based optimization, join ordering, and massively parallel processing to reduce data movement and improve performance at scale. Unlike traditional databases and data warehouses, which combine storage and query processing, standalone engines such as Trino can join data from multiple systems in place. Query engines also support lakehouse architectures and can provide governed access to live enterprise data for analytics, AI, and agentic workloads, potentially avoiding separate ETL or retrieval pipelines. Starburst, built on Trino, adds enterprise features including broader connectors, centralized governance, security, and enhanced optimization, while organizations evaluating query engines should consider performance, connector coverage, federation capabilities, governance, and operational complexity.

Data gravity describes how large datasets become costly and difficult to move because of transfer time, bandwidth constraints, egress fees, latency, and regulatory requirements, drawing compute, applications, and services toward the locations where data resides. The piece argues that enterprise data environments will remain permanently heterogeneous due to operational systems, specialized workloads, organizational autonomy, data-residency laws, and the coexistence of lakes and warehouses, making complete consolidation impractical and potentially increasing vendor lock-in. Rather than centralizing all data, organizations should move processing closer to data and use federated access to provide governed, cross-system analytics and AI context without unnecessary transfers. It presents Starburst’s Trino-based platform as a governed access layer that queries distributed sources, pushes processing to source systems, and can combine lakehouse capabilities with federation. The recommended approach is to inventory where data is large, restricted, or siloed; establish a federated governance layer; and create curated data products with consistent metadata, security, and business definitions so users and AI systems can access reliable enterprise context regardless of where data is stored.