January 2022 Summaries

The concept of "data as a product" encompasses various interpretations, primarily focusing on applying key product development principles to data projects, such as identifying unmet needs, enhancing agility, ensuring iterability, and promoting reusability. This approach encourages data teams to adopt a product mindset, aiming to improve decision-making and create scalable, valuable data solutions within organizations. Different perspectives highlight the importance of integrating tools, processes, and people, with some emphasizing automated data delivery to stakeholders and others focusing on a comprehensive company-wide mindset. The ongoing debate reflects diverse opinions on the best practices for implementing data as a product, underscoring the need for data teams to advocate for the strategic importance of data and find effective ways to integrate it into organizational processes.

The article discusses how to manage a modern data stack as code using Terraform, an open-source project that provides infrastructure-as-code and implements change management of the modern data stack. It explains how to create and maintain dev, test, and prod modern data stack deployments with Fivetran's Terraform provider. The article also explores different ways to manage a modern data stack as code and segment it into dev/test/prod environments using Terraform. These include change management with dev/test/prod environments, implementation options such as separation at the logical level or physical level, and differentiation within modules, workspaces, or repositories. The article concludes by emphasizing that regardless of how and why a modern data stack needs to be split up, doing so with Terraform allows these changes to happen as code, where scale can be achieved effectively and managed efficiently.

In 2021, Fivetran made significant strides in improving its data integration platform. The company added 40 new connectors, bringing the total to over 180, and deployed 14 new pre-built data models. Security enhancements included the launch of a new plan called Fivetran Business Critical, designed for businesses requiring high levels of security and data governance. Other highlights include advanced role-based access control (RBAC), automated user provisioning, universal column masking, and partnerships with over 150 companies in the Fivetran partner ecosystem. The company also hosted its first Modern Data Stack Conference, featuring speakers from major tech companies and insights on managing data.

Fivetran has introduced universal column masking for all its connectors, including SaaS applications, services, files, events and functions. This feature helps protect sensitive data by making it inaccessible to users querying it in the destination. It works through two methods: column blocking, which fully blocks a column from entering a destination, and column hashing, which anonymizes data before it enters a destination. By employing column masking for sensitive data, businesses can meet data privacy compliance requirements such as GDPR, PCI and HIPAA, improve their security hygiene and ultimately protect their brand reputation.

Fivetran has introduced a new data model that simplifies brand awareness analysis by aggregating engagement metrics from major social media platforms such as Facebook, Instagram, Twitter, and LinkedIn into one table. This allows marketing teams to easily compare metrics across different profiles and identify what's working and what's not. The data model also enables seamless combination of data from multiple accounts on the same data connector for easy reporting. Additionally, Fivetran offers a data model for ad reporting that rolls up ad spend, clicks, and impressions for major ad services, allowing side-by-side comparison with social media engagement metrics to evaluate overall performance.

Fivetran has announced new features for users of both Fivetran and dbt Core, an open-source transformations tool popular in the data analyst community. Integrated scheduling, available from February 1, allows data teams to automatically trigger dbt Core model runs immediately after a Fivetran sync completes, reducing data latency and saving on unnecessary compute costs. Data lineage graphs, coming later in 2022, provide users with a holistic understanding of their ELT data pipelines. These features aim to support the open-source community in getting the fastest, most reliable data where it needs to be.

The article discusses the challenges of integrating data stacks and business systems during mergers and acquisitions (M&As). It highlights how modern cloud-based technologies can help organizations overcome these challenges, citing examples like Salesforce's acquisition of Slack. The article also delves into the data challenges faced by merging organizations, such as centralizing terabytes of data from various sources and streamlining operations to eliminate redundancies. It mentions how full-service managed data integration solutions can help in connecting disparate data sources quickly and efficiently without diverting engineering resources or adding operational complexity. The article concludes with an example of Autodesk Construction Services, which unified its data architecture after multiple acquisitions using a modern data stack consisting of Fivetran, Snowflake, and dbt.

The article explores four distinct methods for syncing data from a Snowflake data warehouse to Segment, aiming to enhance digital marketing operations such as user persona enrichment and advertising. The first method involves using Segment's cloud-app source, where data is transferred from Snowflake to an AWS S3 bucket before syncing to Segment, leveraging the flexibility of event and object cloud-app sources. The second method utilizes Segment’s HTTP tracking API, allowing direct data tracking from websites or applications to Segment through server-side sources and various programming languages, ideal for batch data imports. The third method incorporates Segment’s Python Analytics Library to send data requests, offering efficient operation through internal queuing and asynchronous batch processing. The final approach, reverse ETL with Fivetran Activations, provides a comprehensive solution for syncing data not just to Segment but to multiple destination applications, featuring built-in scheduling and robust support, making it suitable for long-term organizational needs.

Fivetran introduces a data model that combines Shopify and Klaviyo data, offering more comprehensive insights into customer behavior. While native integration between the two platforms syncs some metrics, it lacks details on campaign conversions and can lead to discrepancies. The new data model uses email addresses and last-touch attribution to join data from both connectors, providing a complete picture of the customer journey. Fivetran's pre-built models do 80% of the data modeling work, allowing users to create customized analyses tailored to their business needs.

Fivetran has joined Prefect's Premier Partnership program, integrating its data connectors with Prefect's open-source orchestration platform. The partnership aims to simplify and automate data pipelines for users. Prefect is a dataflow automation tool that uses Python to define tasks and flows, allowing for efficient management of modern data stacks. FivetranSyncTask, developed by Fivetran and Prefect, enables the use of Fivetran connectors within Prefect's automation framework. This integration streamlines the extraction and loading processes, allowing data engineers to set up their modern data stack faster and scale quicker while maintaining control over data accuracy and recency.