January 2025 Summaries

Fivetran is a software company that provides change data capture (CDC) technology for integrating data from various sources into cloud-based data warehouses. To measure the performance of its system, Fivetran designed a benchmark to simulate a high-volume database workload and ran it on common relational databases like Oracle, PostgreSQL, SQL Server, and MySQL. The benchmark showed that Fivetran can deliver fresh data into Snowflake in under 17 minutes, even under sustained high throughput, making it suitable for near real-time data integration. The company plans to use this automated test suite internally to ensure consistent performance, extend the benchmark framework to more system configurations, and share the results publicly.

Fivetran has integrated Anthropic's Claude AI models, including the Claude 3.5 series (Sonnet and Haiku), into its data activation platform, alongside existing models GPT-4o and GPT-4o Mini. These models vary in performance, with Claude 3.5 Sonnet excelling in complex tasks like coding and graduate-level reasoning, while Claude 3.5 Haiku offers a balance of speed and accuracy at a lower cost. GPT-4o is noted for its consistency across various tasks, making it suitable for content generation and analysis, while GPT-4o Mini provides cost-efficiency for high-volume, straightforward tasks. Each model's suitability depends on balancing performance with cost, and Fivetran suggests using different models for specific applications, such as e-commerce product categorization, customer support automation, and sales operations, to optimize both performance and budget.

2025 is poised to be a pivotal year as organizations worldwide position generative AI as a top strategic priority, with 83% of CDOs and data leaders making it a key focus. This growing emphasis on AI mirrors broader data trends: Data modernization was the top area of investment in 2024, and 82% of organizations with advanced data and analytics maturity saw consistent year-over-year revenue growth. Experts from Fivetran, Databricks, Snowflake, dbt Labs, and Hakkoda share their insights into the key trends and predictions for data and AI in 2025, including the rise of open data lakes, real-time data observability, knowledge graphs, governance, RAG technology, utility compute, and centralized data ecosystems. These predictions highlight the growing importance of tools and capabilities such as data lakes, query engines, guardrails, retrieval-augmented generation, utility compute, and AI governance, which will enable businesses to unlock valuable insights from unstructured data, drive smarter decisions in supply chain operations, and ensure accurate and reliable AI deployments.

Fivetran has introduced a new benchmarking system to measure the performance of its data pipelines, enabling users to optimize their usage and observe the results of pipeline sync performance improvements. The benchmark focuses on replication from operational and transactional databases to data warehouses and data lakes, with an emphasis on loading from common OLTP relational databases like Oracle, PostgreSQL, SQL Server, and MySQL. A bulk historical load benchmark has been set up to evaluate throughput, while a TPROC-C workload will be used to assess day-to-day latency of high-volume database connections. The benchmark results show that Fivetran can import large datasets at speeds exceeding 500 GB/hour, demonstrating significant performance improvements over the past year, including expanded parallelism and optimized data load patterns. These advancements enable faster time to value for customers, allowing them to access their data when and where they need it.

The current state of data security in 2025 is characterized by a growing reliance on digital transformation, resulting in unprecedented risks and the need for innovative solutions. Organizations face significant challenges, including high costs of data breaches, poor data quality, and stringent compliance requirements. Traditional approaches to securing data are often ill-equipped to meet these demands, with legacy ETL systems being particularly vulnerable to security risks. DIY data ingestion is also a growing concern, as it can lead to costly and fragmented pipelines that introduce risk. However, modern solutions like Fivetran's Hybrid Deployment offer a cutting-edge approach to secure, scale, and simplify data integration, providing full control and security, seamless management, and flexibility and simplicity. By combining robust security, centralized management, and unparalleled flexibility, Fivetran empowers businesses to securely and efficiently scale their data operations, driving growth and competitiveness in the face of mounting data risks.

Companies are facing massive breaches due to increased scrutiny from regulations like the EU-U.S. Data Privacy Framework and the American Privacy Rights Act, which requires secure data handling more than ever. Modern enterprises handle a large scale of data from various sources, making it challenging to safeguard valuable information. DIY data integration is inherently complicated and engineering-heavy, with high potential for creating security weaknesses. Automated data integration offers a technological solution to address labor scarcity and the challenges of security and governance for data in transit. Hybrid deployment, a model that separates the mechanics of data movement from control plane processes, enables secure automation of on-premises data integration while maintaining sensitive data within its originating environment.

With the growing importance of AI, there are now unmatched opportunities to gain value from data. The greatest challenge a CIO faces is implementing a digital transformation program to become a data-driven enterprise. No matter how clever an algorithm is, it can’t accomplish anything without a solid foundation of data built on automated data integration and infrastructure capable of supporting a wide range of data formats. AI plays a significant role in digital transformation by helping integrate and unify data, managing data quality, enabling data governance and contributing deeper analytics. This solution is illustrated by the reference architecture that shows the progression of data from all sources to storage and analysis. AI enhances data management at many stages, including data integration and unification, data quality management, data governance and compliance, data analytics and insights, intelligent automation, and automating repetitive tasks, ultimately empowering CIOs with advanced capabilities to tackle data management challenges and maximize the value of their data assets.

Data governance and security have emerged as major challenges for C-suite leaders pursuing AI, with 44% of executives citing these issues as major obstacles in their AI efforts. Robust security and governance frameworks are crucial to manage data across interconnected systems, particularly in regulated sectors like government and financial services. Centralizing operational data into data lakes or warehouses can pose non-trivial risks in setting appropriate data access controls for downstream AI use cases. Ensuring the right tools and controls are in place to monitor and manage both data and its usage is a new challenge for most enterprises. Data used to train models requires additional security measures to protect it, particularly when using third-party AI services. Failing to manage these risks can result in security breaches, financial loss, and damage to an organization's reputation. Implementing robust data governance frameworks is essential to mitigate these risks, including setting clear policies for data usage, access, and storage, and enforcing them consistently across all AI initiatives.