September 2024 Summaries

Spark has played a significant role in the development and excitement around Iceberg, leading to strong support for Iceberg in the Databricks ecosystem, which is primarily built on Spark. Databricks' Unity Catalog has enhanced governance capabilities and now natively supports Iceberg as part of its catalog, allowing integration with the UniForm feature to generate Iceberg metadata tables alongside Delta Lake tables without conversion. However, UniForm compatibility has limitations, such as issues with deletion vectors and lack of support for Iceberg Views. Databricks recently introduced OAuth support for the Iceberg Catalog REST API, enabling direct access to Iceberg tables, though writing capabilities remain unsupported. Challenges exist with querying nested namespaces due to Databricks' handling of the unit separator, indicating the early but rapidly evolving state of the technology. In June 2024, Databricks acquired Tabular, an Iceberg data catalog, signaling potential future synergies in their offerings, though Tabular is no longer accepting new signups.

A new MIT Technology Review Insights report reveals that 64% of C-suite executives prioritize data readiness for AI success, but face challenges in building the necessary data foundation. The biggest pitfalls include data integration and pipelines, with 45% of respondents citing data integration as their top challenge. Legacy DIY methods are often used for enterprise data pipelines, leading to average losses of $406 million per year. Automated, reliable, and secure data integration is crucial for trustworthy AI, but many organizations struggle with this. The report highlights the importance of a strong data foundation for GenAI, as well as the need for a data quality mitigation strategy like retrieval-augmented generation (RAG) to incorporate proprietary business data. DIY data pipelines do not scale and can become costly liabilities over time. Modern, automated solutions like Fivetran offer built-in schema change support and automatic propagation of data source changes, reducing the operational burden and improving AI performance.

Data ingestion costs refer to expenses associated with loading data into storage systems for processing and analysis, impacted by factors such as data volume, frequency of transfer, complexity, and fees. Ingestion can consume up to 30% of compute costs in a data warehouse. Fivetran's Managed Data Lake Service simplifies data lake management, automatically converting customer data to popular open formats before landing it in the data lake. The service covers ingestion costs as part of its pricing model, making it cost-effective for businesses. Organizations with an architecture containing one or more data warehouses stand to gain the most from moving to a data lake architecture, potentially leading to substantial savings.

GenAI technology is set to revolutionize businesses by delegating decision-making authority to AI in unprecedented ways. Key strategies for AI preparedness include data accuracy, governance, and knowledge graphs. Data curation plays a critical role in AI success, with the gap between proof-of-concept and production-ready systems being significant. Businesses must prioritize data quality and build a solid foundation to support AI's growing role in decision-making. Knowledge graphs represent a major opportunity for businesses to make the most of AI by enabling machines to understand and organize complex information. In 2025, predictions include AI workloads resembling business intelligence, knowledge bases becoming mainstream, and purpose-specific AI models outperforming general ones. Data will be key in ensuring better, faster insights that can take organizations to the next level.

Data is crucial for AI applications, with unstructured data accounting for about 80% of all data. Scalable vector databases like Milvus are essential for efficiently storing and searching across an organization's information. Fivetran's Milvus destination simplifies the process of ingesting data from various sources into Milvus, enabling AI applications to access diverse datasets and adapt to changes in data sources. This integration makes it easier to develop AI workflows by ingesting source data from a wide range of databases/warehouses and business apps to Milvus vector database.

Fivetran introduces Hybrid Deployment, a new deployment option available on their Enterprise and Business Critical plans. This innovative solution allows users to run sensitive or security-restricted data pipelines within their own secure network or VPC, giving them full control over their data without the complexities of additional software maintenance. The Hybrid Deployment offers benefits such as full control and security, seamless management, and flexibility and simplicity. It enables enterprises to centralize and secure all their data, regardless of its sensitivity or security, and can lead to significant economic benefits.

The average Snowflake customer spends about $300,000 per year on the platform, with 90% of this being queries. Data warehouses are primarily used as ETL tools, and most queries are small, scanning around 100 MB. Massively parallel processing (MPP) systems like Snowflake and Redshift distribute data across multiple computers for query execution, but specialized engines can be more efficient for specific workloads. The future of data warehouses may lie in data lakes, which separate storage into vendor-neutral formats and allow multiple specialized execution engines to interact with the same data, improving efficiency and user experience.

Generative AI is increasingly important in today's data-driven world, but many organizations struggle with data integration, leading to inefficiencies and missed opportunities for scalability and growth. CIOs play a crucial role in driving business success through effective data strategies that include standardizing and maximizing data accessibility to pave the way for scaling AI. Building a strong data foundation is essential for meaningful AI success, with key steps including managing costs by centralizing platforms and controlling infrastructure sprawl, fostering a culture of effective data stewardship and governance, and aligning data strategies with impactful AI/ML use cases. By doing so, CIOs can create a solid foundation for AI initiatives that drive more effective, innovative, and responsible data use throughout the organization.

The text discusses the challenges and potential of operationalizing AI within businesses, emphasizing the need for AI to be trusted, scaled, and impactful to truly revolutionize business operations. It highlights the limitations of current AI tools, which often focus on individual productivity rather than large-scale business transformation. The key to successful AI integration lies in operationalized use cases that are centrally managed by ops teams, leveraging their understanding of governance and workflows. The text outlines various AI tools, such as Snowflake Cortex and Salesforce Einstein, which are currently being used to enhance data processes, automate repetitive tasks, and support complex analyses across organizations. Additionally, it emphasizes the importance of good data quality and strategic implementation to maximize AI's impact, encouraging businesses to experiment with prompts and share successful strategies to foster broader AI adoption.

Data integration is a top priority for enterprise executives, with 82% of senior executives considering scaling AI as a top priority. However, this ambition is hindered by the longstanding practice of maintaining separate data architectures for batch and streaming use cases. Separate data architectures pose serious challenges such as expensive duplicate provisioning of infrastructure and engineering time, conflicting versions of truth, and redundancy in governance and security efforts. The emergence of governed data lakes combines the capabilities of data warehouses and data lakes, allowing a single data stack to support both batch and streaming use cases. A unified, automated data platform will be essential for supporting generative AI as new use cases emerge that leverage data from novel combinations of sources. Automation can sidestep the problem of mastering different engineering languages and paradigms for batch and streaming use cases by enabling teams to assemble a data architecture using easy-to-use, off-the-shelf tools.

Snowflake's introduction of Managed Iceberg Tables allows users to construct an Iceberg data lake within Snowflake, utilizing a new object called an EXTERNAL VOLUME for interacting with Iceberg tables in object storage. This feature necessitates setting up IAM controls for Snowflake's access and includes the creation of Iceberg tables within these external volumes, though it does not support direct conversion of existing tables. Users may need to adjust API data types when migrating tables, such as changing TIMESTAMP formats. Snowflake-managed Iceberg tables come with a proprietary catalog that is limited to JDBC connections, restricting external services to read-only access. The catalog includes a CATALOG_SYNC feature that synchronizes changes made in Snowflake with an external catalog in Polaris, which supports Iceberg REST and potentially allows connections between multiple Snowflakes. Despite being relatively new, this managed catalog might eventually be integrated into Polaris, which could expand its utility for Iceberg architectures.

Last week, Snowflake's native Managed Iceberg Tables were examined, revealing some limitations, but a new approach called Polaris has been announced as an Apache incubated Iceberg Catalog. Polaris, which implements the Iceberg Catalog REST API for improved cross-compatibility, is available as a hosted service for Snowflake customers and as an open-source project for self-hosting. It introduces a simpler access control layer than other data warehouses, with catalogs acting as top-level containers for table sets grouped into namespaces, offering a more flexible structure than the traditional three-tier naming convention. Catalogs can be internal, managed by Polaris, or external, managed by another Iceberg catalog provider. The access control system is based on Principal Roles and Catalog Roles, with privileges applied at the catalog level. Setting up Polaris involves creating a catalog, establishing connections, and deciding on using Snowflake or a separate process like Spark for managing Iceberg tables, with specific steps depending on the chosen method.

Fivetran is committed to data protection and privacy compliance across over a hundred global regimes, incorporating "privacy by design" and "privacy by default." Key security features include column blocking and hashing, encryption, metadata logging, and GDPR-compliant measures. The company uses the AICPA/CICA Privacy Maturity Model to measure its privacy program's success. Fivetran also handles Data Subject Access Requests (DSARs) and cross-border transfers while maintaining a list of global subprocessors and data residency options across 20+ major cloud regions worldwide. Regular privacy assessments, data mapping, and retention efforts ensure ongoing compliance with international privacy laws.

Bear Cognition is a business intelligence company that uses cutting-edge solutions to drive efficiency and stronger bottom lines for clients in industries like marketing, hospitality, retail, and supply chain and logistics. The company faced challenges with data management before adopting Fivetran, which significantly improved their operations by enabling seamless integration of various data sources, reducing data engineering time, and expanding market reach. Bear Cognition's partnership with Fivetran has transformed its data integration process, providing faster, more flexible, and highly scalable data solutions to clients.

Fivetran offers three deployment options for data movement, catering to various security requirements and use cases. These include a fully cloud-based SaaS option, a hybrid option that separates the data and control planes between the cloud and your environment, and a self-hosted option within your network or VPC. The company's solutions aim to provide secure and efficient data integration processes for business insights, predictive analytics, and security purposes.

Iceberg, initially developed by Netflix and now part of Apache, is an open-source table format designed to address high-scale data challenges by separating storage from compute, allowing reliable data access across multiple concurrent readers and writers. It contrasts with traditional file formats like CSV and Parquet by representing a database table with a cluster of files for data and metadata, often written in Parquet. Iceberg's design supports scalability, cost optimization, mixed-compute support, and open format preference, enabling users to avoid vendor lock-in and integrate with various tools. Its key feature is the separation of storage and compute, allowing for customized storage choices and dynamic compute allocation, which supports data sharing and a Zero Copy/Zero ETL approach. While Iceberg offers numerous advantages, its catalog, which acts as an authority for updates and ensures consistent metadata, poses challenges due to the current limited support across different platforms. Despite these limitations, the ecosystem is progressing towards broader support, and Iceberg's potential benefits make it an attractive option for companies dealing with large-scale data needs.