July 2021 Summaries

Fivetran and Hightouch have partnered to turn your data warehouse into your customer data platform. While Fivetran loads data from 180+ popular tools into your warehouse, Hightouch sends data from your warehouse or lakehouse to the tools that your business teams rely on through a process called reverse ETL. This partnership aims to activate your data and power the modern data stack by using the data warehouse not just as an analytics layer but also as a modeling layer that sits at the center of your data stack, allowing any team to keep their tools synced. Companies are already benefiting from this technology, such as Retool increasing its reply rate by over 32% and Zeplin improving sales productivity. Hightouch is built for data teams with unique features like a live debugger and version control via Git.

The Modern Data Stack Conference will feature the inaugural Modern Data Stack Awards, honoring innovative data teams across seven categories including Data Hero and Transformation for Good. Award winners will be announced on the second day of the conference. Businesses impacted by the modern data stack can submit nominations until September 9th, with a panel of industry experts selecting the winners.

Gartner recognizes Fivetran as the "vendor of choice" for enterprises looking to modernize their data infrastructure. The company's mission is to make data access as simple and reliable as electricity, leveraging automation and cloud data platforms to centralize and analyze data easily for customers regardless of the number of sources they have. Fivetran has doubled its revenue between 2019 and 2020 and continues to prosper in the cloud-native data ingestion space. The company's growth is driven by product capabilities promoting ease of use, such as fully managed, self-healing data pipelines, minimal infrastructure management, and automated normalization of data.

Data warehouses are a type of database designed specifically for analytics, offering faster access to aggregated information compared to traditional databases. While both use relational structures, databases store data by rows and are optimized for transactional processing (OLTP), while data warehouses store data by columns and are optimized for analytical processing (OLAP). Data in a database is updated frequently, representing real-world transactions and events, whereas data in a warehouse is updated in batches for analysis. Using the right tool for the job, such as a data warehouse for analytics, can significantly improve efficiency and speed up results.

HubSpot is a popular CRM platform that provides valuable engagement and behavioral data within its various solutions or "hubs." However, combining this data to generate a holistic customer view can be challenging. Fivetran's HubSpot connector aims to address this issue by centralizing the data into the destination of choice and preparing it for analytics. This allows businesses to analyze the entire customer lifecycle from marketing engagement through product trials, deal stages, and beyond. By integrating HubSpot data with Fivetran and a modern data stack, companies can enrich their data across different departments and gain visibility into the entire customer lifecycle.

The second annual Modern Data Stack Conference of 2021 is set to take place on September 22nd and 23rd, featuring over 50 data industry leaders, practitioners, and visionaries discussing the transformation of businesses through next-generation analytics. Key topics covered will include the future of the modern data stack, best practices for building and maintaining a data stack, and how to create a data stack tailored to specific company types and growth stages. The event also includes opportunities for networking and community building among data professionals. Registration is now open, with more details on speakers and sessions to be released soon.

The article discusses the importance of ETL (Extract, Transform, Load) pipeline security for safeguarding customer's data and complying with regulations. It highlights how Fivetran ensures data security by protecting physical infrastructure, using web application firewalls, intrusion detection systems, and anomaly-based security monitoring systems. The company also maintains several industry security standards such as ISO/IEC 27001, SOC 2 Type 2 audit, PCI DSS Level 1 validation, EU 94/95 privacy rules, US/EU Privacy Shield program, GDPR rules, and HIPAA requirements. Furthermore, Fivetran has an incident response program in place to handle security incidents effectively. The article concludes by stating that customers can trust Fivetran with their data as it takes all necessary precautions for ETL pipeline security.

The article discusses the benefits of using in-warehouse machine learning services to create a modern data science stack. It highlights that these services remove silos and duplicated work for data analytics and data science teams, making models closer to the data they are training with and using for predictions. This shift from model-centric AI development to data-centric AI development is facilitated by having models stored in a data warehouse, which allows their predictions to be obtained via SQL queries. The article also mentions that these services can help avoid training-serving skew and make it straightforward to compose different steps of the machine learning process into a data pipeline. It provides an overview of BigQuery ML and Redshift ML, two in-warehouse machine learning services offered by Google Cloud Platform and AWS respectively. Additionally, it mentions that Snowflake can integrate with various machine learning tools like Sagemaker and Databricks to create a modern data science stack. The article concludes by emphasizing the importance of having a centralized location for all data-related tasks in an organization interested in performing both data analytics and data science.

Navigating the expanding realm of data, particularly in Snowflake, presents challenges in exporting data into CSV files for various purposes such as importing into other tools, moving across cloud platforms, or visualizing data for analysis. The text discusses three primary methods for exporting CSV files from Snowflake: using BI tools like Mode and Metabase for direct downloads, employing the COPY command to save data to cloud storage solutions such as AWS, GCP, or Azure, and utilizing SnowSQL, a command-line tool, to execute queries and manage data. Additionally, the text highlights Fivetran Activations as a sophisticated solution for reverse ETL, syncing data from Snowflake to various destinations like Salesforce and Hubspot, emphasizing that the choice of method depends on the data size and destination.

Data transformations are crucial in connecting and making the most of collected data. They help businesses produce data models to answer critical decisions by joining, filtering, and aggregating data. Common use cases include reporting, BI, visualizations, dealing with big data, advanced analytics, machine learning, and artificial intelligence. Some data needs to be transformed regardless of use case, such as global data, removing personal identifying information (PII), and cleaning dirty data. Modern transformation methods are increasingly important in the world of cloud computing and data complexity. A modern ELT approach like Fivetran Transformations ensures that data is modeled for analytics to maximize its impact across businesses.

French home insurance company Luko has leveraged its modern data stack to develop a new business model that focuses on preventing problems and reducing claims, rather than profiting from them. Modern data collection and analysis are critical to Luko's approach, with the company using various services that rely on data for their effectiveness. The use of a centralized data warehouse and efficient transformation and visualization tools has enabled everyone in the company to have easy access to reliable and up-to-date data, improving decision-making across the organization. This modern data stack has allowed Luko to grow, scale, and innovate, with the combination of Fivetran connectors and dbt being particularly useful.

dbt (data build tool) is a SQL-based transformation and modeling tool developed by dbt Labs that automates the data transformation process. It allows data professionals to iteratively build data models, create relationships and dependencies between them, and materialize those models as tables and views in a data warehouse. The YAML files are used for project configuration, curating a data dictionary, and applying data integrity tests. dbt is not an ETL tool but works with Fivetran to provide end-to-end ELT processes. It integrates directly into Fivetran, allowing users to orchestrate cleaning, testing, transformation, modeling, and documentation of data sets.

Marketing mix modeling is a data-driven technique used to evaluate the impact of various marketing channels, such as TV, radio, and newspapers, on a company's sales or conversions. With the advent of online marketing and big data, marketing is increasingly viewed as a mathematical discipline, where methods like marketing mix modeling offer significant opportunities for data science and machine learning applications. This modeling technique is distinct from other attribution models as it provides a comprehensive measure of the impact of marketing channels over different time periods. The article provides a detailed guide on building a marketing mix model using Python, beginning with exploratory data analysis to assess relationships between variables, followed by constructing a model using ordinary least squares regression. The goal is to identify high ROI channels and optimize marketing budgets while predicting future outcomes. The effectiveness of the model is evaluated using metrics such as the adjusted R-squared and p-values, indicating the significance of each channel, with practical insights into how well the model predicts sales based on marketing spend.

The article discusses how Software as a Service (SaaS) providers can build robust data replication APIs. It emphasizes the importance of providing an incremental update mechanism to ensure fresh and accessible data for users. Two main approaches to incremental updates are presented - change logs and modified timestamps, each with its own trade-offs. The article also highlights the significance of making full syncs straightforward by ensuring all data is provided quickly and efficiently. Proper documentation is considered crucial in avoiding confusion and misunderstandings when working with API endpoints. By providing a high-quality API, SaaS providers can empower their users to integrate and leverage their data effectively, leading to increased success for both parties.

In a recent presentation at the Modern Data Stack Conference EMEA, data architecture expert Nas Radev shared insights on building a modern data stack from scratch. The components of such a stack include a data ingestion solution, a data warehouse or data lake, a transformation tool, and a business intelligence (BI) tool. To successfully build this stack, businesses must first identify their data sources and choose the right ingestion tool. Next, they should plan for transformation and select a suitable data warehouse, possibly using an open-source tool like dbt to simplify the process. Finally, they need to pick a BI tool that aligns with their specific use case. Throughout this process, Radev emphasizes the importance of maintaining high data quality across all components to ensure success.

Amazon Redshift now supports native integration with select AWS partners, including Fivetran. This integration accelerates data onboarding and creates valuable business insights in minutes. Users can launch a 14-day free trial of Fivetran directly from the Amazon Redshift console, where configuration details are automatically pre-populated. The integration enables Fivetran to extract application and other source data, load and centralize all the data into Redshift, and then transform it within Redshift for quick analysis and sharing.

Reverse ETL involves transferring data from a data warehouse back into operational tools, such as moving data from Postgres to Google Sheets, which is particularly useful for small teams needing to add notes and share information seamlessly. Using Python, libraries like psycopg2 and pandas facilitate reading data from Postgres, while gspread and gspread-dataframe enable writing this data to Google Sheets. Setting up access involves retrieving credentials from Google Cloud Platform and enabling the Google Drive API. Although the process of connecting Postgres to Google Sheets is straightforward, scaling these small scripts for regular use can lead to technical debt, necessitating the automation of tasks and the creation of new data structures as needed. To address these challenges, using scalable integration solutions, such as Fivetran Activations, can simplify the synchronization of customer data from the warehouse to operational tools, reducing the need for custom code and allowing for more efficient data management.