June 2023 Summaries

The text discusses the importance of successful cloud migration for businesses, citing a 25% average revenue growth as a result. However, navigating the challenges of cloud migration is crucial to realizing these outcomes. Industry experts identified key obstacles such as cost visibility, data governance, strategy gap, and mindset/multi-cloud thinking that organizations face during cloud migration. To overcome these challenges, organizations should focus on building robust technical infrastructure, leveraging intelligent analytics tools, and creating valuable data products that drive business value. Establishing a strong definition of data quality, implementing data governance frameworks, and aligning cloud migration strategy with overall business objectives are essential to overcoming the obstacles and harnessing the full potential of the cloud.

Fivetran has been named Databricks 2023 Innovation Partner of the Year, recognizing its transformative impact within the Databricks ecosystem through successful product integrations such as Unity Catalog, Partner Connect for Non-Admin and Databricks SQL Serverless. This award builds on previous recognitions from Databricks, highlighting how Fivetran's solutions work together to help organizations garner the most value out of their data. By combining Fivetran's data pipeline expertise with Databricks' analytics capabilities, businesses can seamlessly access and analyze their data in real-time, driving informed decision-making and innovation. The partnership enables data teams to focus on extracting insights from data rather than spending time on mundane tasks, empowering organizations to leverage the full potential of their data. Fivetran's integration with Databricks has proven a game-changer for businesses seeking to leverage unified data analytics platforms, such as Conde' Nast and YipitData.

Fivetran's analytics engineering team has developed a robust library of open-source dbt data models that help customers turn raw connector data into analytics-ready tables. These data models are used by over 1,000 new projects each month and can be accessed by anyone. The data models extend the connector entity relationship diagrams (ERDs) to create these analytics-ready tables, which perform foundational work that occupies an analyst's time. They identify relevant entities in the ERD and perform transformations, ultimately producing output tables needed for analytical workflows. Fivetran's data models facilitate the process of generating a general ledger from QuickBooks schema by identifying necessary tables, creating intermediate models, and consolidating data to provide a comprehensive understanding of transactions. The team maintains and updates these data models based on customer feedback, ensuring they include all transaction types and rework is not needed. With Fivetran's data models, customers can accelerate their time to insight and spend less time building and maintaining ELT pipelines.

Data transformation is an integral part of the ELT (Extract-Load-Transform) process, which involves cleansing, modeling and manipulating raw data to prepare it for analysis. This step is crucial as raw data is not prepared for reporting or generating insights. Transformations might include removing duplicate rows or entries, joining disparate tables to generate KPIs, normalizing data, creating calculations or business logic classifications, among others. Automating the transformation process and orchestrating it in harmony with data load can help reduce tech stack complexity, data latency, and computational costs. Fivetran Transformations for dbt Core is a solution that automates the transformation process and manages it within one platform, providing increased visibility and governance.

Microsoft Dynamics 365 is a cloud-based suite of business application platforms designed to modernize enterprise resource planning, operations, and customer relationship management. Harnessing these applications can enhance agility, achieve operational excellence, and deliver exceptional customer experiences. To break down data silos, conduct predictive analysis, and facilitate cross-functional decision-making, organizations can leverage the practice of data ingestion. Fivetran offers out-of-the-box connectors for Microsoft Dynamics 365, including Finance and CRM, to centralize data, eliminate data silos, and unlock actionable insights. The connectors provide access to raw underlying data directly from the source, enabling unified analysis and leveraging industry standards. By using Fivetran, organizations can empower their teams to make informed decisions that drive success.

The main reasons to modernize infrastructure are to enable centralization of data at infinite scale, with modest upfront capital outlays, and to complement other cloud-based analytics technologies. Modernizing infrastructure improves flexibility in terms of compute and storage, reduces costs, and lowers engineering workloads. The advantages of cloud-based data infrastructure include ease of use, scalability, cost control, and interoperability with newer technologies. Centralizing data is the first and essential step in enabling actionable use of data, especially for democratizing data and building systems that monetize data. Modernization can take place along several dimensions, including moving from on-premise to cloud, switching from ETL to ELT-based architecture, migrating destinations, and upgrading data movement capabilities from batch to real-time.

Fivetran CEO George Fraser and Snowflake CEO Frank Slootman discussed the current state of enterprise data strategy, highlighting its importance for achieving financial efficiency in a challenging market, centralizing ERP data to unlock greater value from an organization's full data landscape, and leveraging low-latency enterprise data to drive faster, more informed decisions. The combination of these factors will significantly influence the future of enterprises as they develop their data strategy. By partnering with Fivetran, Snowflake is helping customers move and leverage all their data easily, without considering how it reaches its destination, thereby removing risk and friction from data pipelines and enabling enterprises to make educated decisions in real-time.

Customer Data Platforms (CDPs) are software tools designed to collect and organize customer data from various sources to create unified profiles, which can be used to personalize marketing and improve customer service. Despite their potential, CDPs often fall short of being the complete solution they claim to be due to issues like data quality, vendor lock-in, and not truly serving as a "single source of truth." Instead, businesses with established data infrastructures might find greater value in leveraging their data lakes, which store raw data in its native format, and using tools like Fivetran Activations to create a "composable" CDP. This approach allows businesses to maintain a consistent source of truth for customer data, avoid vendor lock-in, and adapt their systems flexibly to changing needs, using their data lake alongside tools for data transformation and activation. By transforming data and defining customer segments, businesses can sync information to various operational systems, thus enabling personalized marketing and improved customer interactions without the limitations of off-the-shelf CDP solutions.

Fivetran and Matillion are two long-standing data integration providers that cater to businesses with diverse data needs. While both offer low- and no-code pipelines, custom development capabilities, and powerful transformation features, Fivetran stands out for its robust, managed pipelines with multiple change data capture (CDC) capabilities, built-in advanced features for modern workflows, and guaranteed high availability. In contrast, Matillion's approach is more traditional ETL-based, relying on agents, manual configuration of schemas, and a proprietary GUI that may require additional expertise and infrastructure management. Fivetran offers nearly three times as many connectors, many of which are feature-rich, and its agentless architecture simplifies pipeline setup and scalability without the need for customer-managed infrastructure. With Fivetran, users can set up fully managed pipelines with automated schema maintenance, private connections to cloud providers, column blocking and hashing, high-volume replication, and transparent pricing, making it an attractive choice for organizations seeking a forward-looking data movement platform with clear documentation and advanced security features.

Fivetran has partnered with Microsoft to help organizations prepare their data stacks for AI adoption, highlighting the importance of reliability in data quality and movement, as well as accessible, flexible storage that can handle multi-modal outputs. To achieve these goals, organizations must adopt a modern data architecture that combines the best of a data lake, data warehouse, and operational data store, leveraging tools like Fivetran's automated and fully-managed service to ensure idempotence and data reliability. With this foundation in place, organizations can unlock the full potential of AI-driven business outcomes and navigate the future with confidence.

The text discusses the challenges of data governance in organizations, particularly when it comes to balancing competing interests between security and legal teams, data producers, and consumers. It highlights the need for technological solutions that can provide both visibility and control over data access and compliance at scale. The solution involves leveraging automation, metadata management, and data catalogs to enable real-time monitoring, audit trails, and programmatic management of data movement. This approach aims to overcome the tradeoff between access and compliance, ensuring that all stakeholders have access to fresh and relevant data while protecting sensitive information.

Data centralization is essential for retail businesses as it enables them to manage and analyze data effectively, providing insights that support business decisions. Retailers need to centralize their data from various sources to gain a comprehensive view of their operations, customers, and products. This leads to optimized internal processes, improved customer engagement, and increased profitability. Data centralization is the first step in digital transformation, enabling data democratization and the building of data solutions. It also enables organizations to modernize their infrastructure, making it more agile, responsive, and innovative. By automating data movement, ensuring reliability, and scaling data operations, retailers can achieve a cohesive strategy for digital transformation that drives business growth and success.

Data teams can justify investment in data infrastructure by creating revenue-driving data practices that are tied to specific business outcomes, such as increasing earnings or controlling costs. For product-led growth, data teams can use experimentation to drive "aha" moments via AB experiments, which can be implemented with minimal infrastructure using simple methods like randomizing users into groups based on the last digit of their user ID. Marketing teams can optimize marketing spend and improve ROI by analyzing lifetime value and payback periods, while sales teams can enrich lead, opportunity, and account data to create more efficient processes and maximize new deal volume and size. By demonstrating value using simple methods and a modest amount of data, data teams can enable CFOs and other gatekeepers to picture the possibilities of making analytics projects repeatable and expanding the scope, scale, and sophistication of their data operations.

Pete Williams, a finance transformation expert and leader at Penguin Random House, discusses managing chaos in data culture transformation by focusing on objectives and seizing opportunities to move forward despite detours. He shares his strategy model centered around the three 'Es' of data maturity: Establish, Enable, and Exploit. In the Establish phase, he focuses on building a strong foundation through breaking down silos, embedding data governance, and providing flexible analytics solutions. This phase laid the groundwork for the Enable phase, where momentum is maintained, and the focus shifts to making work relevant to more people and driving curiosity and digital processes. The goal of Exploit is to push into new territory by exploiting insights and driving business outcomes. Williams' approach emphasizes the importance of data literacy, ownership, and enabling a culture that fosters leadership and innovation.

Apache Iceberg is an open-source table format that provides a number of benefits over alternative data lake file and table formats, including improved query performance, consistency, and accuracy, while also making it easier to manage and evolve data over time. A key component of a data lakehouse stack is the storage layer, which enables organizations to store large volumes and varieties of data types in a flexible and cost-effective manner. Apache Iceberg tables are well-positioned for sustainable development and enterprise adoption, with many contributors from technology companies including Netflix, Apple, Google, AWS, Stripe, Dremio, and others. Data teams can leverage data from the data lake by moving it into the data warehouse using proprietary formats, or relying on data copies to meet Service-Level Agreements (SLAs) for performance. The result is a lot of complexity and management overhead, particularly as requests for access to data in the data lake inevitably increase. A data lakehouse combines the best capabilities of data lakes and data warehouses, enabling organizations to store large volumes and varieties of data types in a flexible and cost-effective manner, while also satisfying a wide range of analytics use cases, including Business Intelligence (BI) and reporting. Fivetran automates the process of bringing data from a variety of sources into cloud data lake storage, providing pre-built connectors for a wide range of data sources, automatic schema detection and mapping, and making it easy to build Iceberg tables. Dremio enables self-service access to Iceberg tables with sub-second query performance, using its semantic layer and query engine based on Apache Arrow, an in-memory columnar format designed for interactive analytics.

The modern data stack has become critical for leveraging AI, as highlighted in Databricks' 2023 State of Data + AI report. The report found that companies are increasingly using SaaS LLM APIs and investing in the modern data stack to train models. However, without effective tools and practices, organizations can block themselves from successfully implementing AI due to issues with data movement and transformation. Utilizing an ELT platform like Fivetran can automate these processes, freeing up resources for higher-value work. The report also highlights the importance of transforming large volumes of raw data into a structured format suitable for training models, with tools like dbt and Fivetran playing a crucial role in this process. Additionally, storing data in a data lakehouse, which combines the flexibility of a data lake and the management methods of a data warehouse, is essential for leveraging AI. Ultimately, the future of AI will depend on how organizations utilize their modern data stack to achieve innovative results.

Fivetran has shifted its stance on data lakes due to changing customer needs, with large volumes of semi-structured and unstructured data becoming more common, making the cost advantages of data lakes more meaningful, particularly for AI and machine learning. Data lakes are increasingly capable in terms of cataloging, governance, and handling structured data, as well as security and regulatory compliance. The functions and capabilities of data warehouses and data lakes are consolidating under a common cloud data platform or data lakehouse, simplifying an organization's data architecture. Automated data movement unlocks the potential of the data lake by streamlining the data pipeline and reducing labor-intensive transformation stages, allowing for more efficient data engineering and data science efforts.

Customer Data Platforms (CDPs) were initially hailed as essential tools for organizing and managing customer data by creating unified customer profiles without heavy engineering involvement, but they have encountered significant challenges and limitations. While CDPs were designed to unify data for marketing teams, they often fall short, offering only partial solutions and sometimes overlapping functionalities with other tools like Data Management Platforms and Customer Relationship Management systems. They frequently fail to provide a single source of truth, necessitating additional business intelligence tools, and lack the flexibility required by diverse businesses. In contrast, modern data warehouses, when combined with Data Activation platforms like Fivetran Activations, can serve as a more effective solution, acting as the central source of truth and enabling data activation across an organization without the need for complex SQL knowledge. This approach allows businesses to leverage existing data warehouses to achieve what CDPs promise, with greater flexibility and without the pitfalls of traditional CDPs.

The importance of diverse data teams cannot be overstated in today's market, where artificial intelligence tools are increasingly impacting the work of employees. A diverse team ensures that data is representative of the customer base, and women play a crucial role in this process. Women in data leadership must increase recognition of their contributions and celebrate their strengths to drive business success. Effective data leaders prioritize team well-being, protect against burnout, and foster a culture of self-praise, emphasizing the performance, impact, and exposure of their teams. As markets change, diverse data sets are essential to reflect this shift, and women in data are instrumental in shaping the future of data-driven decision-making.