February 2019 Summaries

Snowflake is a leading cloud-based data warehouse designed to deliver exceptional performance, concurrency, and simplicity, allowing multiple users to access extensive datasets simultaneously at significantly reduced costs compared to traditional solutions. It operates on a pay-as-you-go model and efficiently manages both structured and semi-structured data. Fivetran complements Snowflake by providing a zero-maintenance data replication pipeline that simplifies the integration of data from various applications and databases into a centralized location, facilitating rapid setup and enabling analysts to derive valuable business insights quickly.

Fivetran has released a new version of their Criteo connector, based on the new Criteo REST API. This allows users to extract data from Criteo and load it into their data warehouse for more comprehensive analysis. By combining Criteo data with other sources like CRM platforms, users can gain insights into their full customer journey and measure ROI on marketing spend. The connector setup is quick and easy with Open Authorization (OAuth) support. Fivetran's documentation provides guidance on setting up the connector and centralizing all data sources for more powerful metrics.

Fivetran has introduced a new connector for Adobe Analytics, allowing businesses to extract and load their analytics data into a data warehouse for further analysis. The connector pulls from the reporting layer of the Adobe Analytics API, providing access to pre-defined reports that help evaluate complex web analytics information. By combining this data with other sources such as ad spend or sales data, companies can gain valuable insights into marketing ROI. Fivetran's documentation provides guidance on setting up the connector and includes schema information.

Charles Wang's article discusses the importance of data warehousing in today's business intelligence landscape. He highlights that while connecting data directly from sources to BI tools can be useful for one-time explorations, it is not a sustainable approach for building dashboards and pursuing data-driven decision-making. Wang outlines several drawbacks to this approach, including the risk of using outdated or poorly structured data, lack of consistency, version control, collaboration, and transparency within an organization. He emphasizes that data warehousing is necessary for any organization whose spreadsheets have become unmanageable, as it provides a common access point with standardized versions and formats. Wang recommends using tools like Fivetran to build data pipelines instead of relying on integrated connectors offered by some BI platforms.

A data lake is a permanent repository of an organization's data in open-source formats like Parquet and blob-stores like S3. While there are good reasons to adopt a data lake, such as reducing vendor lock-in, supporting multiple SQL and non-SQL destinations, and sending the same data to multiple warehouses, there are also some misconceptions about its benefits. Separating compute from storage is not an advantage of data lakes; modern data warehouses can do this more efficiently. Storing raw data in a separate location from curated data is also unnecessary as both types of data can be stored in the same warehouse with different schemas. Additionally, storing semi-structured data in a data lake is not required as modern data warehouses support such data formats. Fivetran offers a fully managed data lake that replicates all data sources to both data lakes and warehouses.

Fivetran services are GDPR compliant, with features designed to help customers meet EU data protection requirements. The company prioritizes customer trust by ensuring data privacy and security through measures such as not retaining data, offering EU servers, providing a robust DPA, encrypting data in transit and at rest, maintaining SOC 2 compliance, supporting HIPAA requirements, and offering column blocking and hashing features. Fivetran's documentation provides further details on their commitment to security and data protection.

Simpson's Paradox illustrates how data can lead to misleading conclusions if not carefully analyzed, as demonstrated through various historical examples. These examples highlight the importance of considering lurking variables and regional or contextual factors that can influence data interpretation. For instance, the 1964 Civil Rights Act voting patterns appeared to show more Republican support, but regional differences revealed that Southern Democrats, due to historical and regional attitudes, largely opposed it. Similarly, gender-based admissions disparities at Berkeley, voter income preferences in the 2016 U.S. election, kidney stone treatment success rates, and rising median incomes despite declining earnings by education level all point to hidden variables affecting outcomes. These instances underscore the necessity of delving deeper into data beyond surface-level analysis to account for external influences and construct coherent narratives, as overlooking such factors can lead to incorrect conclusions.

Charles Wang and Stephen Young demonstrate how to combine AWS Lambda and the new Fivetran REST API to build connector status dashboards. The Fivetran REST API is currently in beta, allowing data engineers to programmatically manage users, groups, and connectors, automating workflows. By using an AWS Lambda function to request a list of all connectors within a group, you can easily assemble a table with the status of every connector syncing to that group's warehouse. This enables near real-time visualization of every connector's status without entering your Fivetran dashboard. The data from this API endpoint is split into three tables: Items, Tasks, and Warnings. A Python script for an AWS Lambda function is provided to help with the implementation. To configure the Lambda connector, follow similar steps as in the Redshift migration script, including setting up IAM policies and roles, creating a function, copying or uploading the Python code, configuring a test event, and adding a connector of the type AWS Lambda.

Fivetran co-founders George Fraser and Taylor Brown discussed recent consolidation in the data pipeline space, including Google's acquisition of Alooma and Talend's acquisition of Stitch Data. They believe that companies will benefit from automations in their data pipelines and a cloud-first ELT approach. For existing Alooma customers using non-Google destinations like Snowflake and Redshift, they may need to look for a new solution in the next year. Fivetran offers its services for free for up to six months for Alooma users needing to switch solutions.

Fivetran CEO George Fraser spoke at MicroStrategy World 2019 about the benefits of cloud data warehousing. He argued that cloud data warehouses are fundamentally different and better than on-premise ones, offering elasticity, speed, and cost advantages. Elasticity allows for dynamic resource allocation, while speed is achieved through column-based OLAP systems optimized for aggregate functions typical of business intelligence. The low cost of cloud data warehousing results from the decline in storage and computation costs over the past two decades. Fraser also highlighted the ELT approach, which reduces complexity and labor costs by automating data pipelines and outsourcing them to vendors like Fivetran. Overall, cloud data warehousing offers a simplified business intelligence architecture with significant cost savings and improved performance.

Fivetran has introduced a Pendo connector in beta, allowing businesses to centralize data on user-product interactions without requiring coding. Pendo helps companies improve product experiences for customers by capturing interaction data and providing personalized guidance and feedback. The documentation provides instructions on setting up the connector and includes an entity relationship diagram (ERD) of the schema. Businesses with a Pendo account can set up the connector to consolidate their data sources for more comprehensive metrics, while Fivetran's support team is available for assistance.

The Fivetran Amplitude connector is now available in beta, allowing businesses to centralize product analytics data from the platform. Amplitude helps companies improve digital products by providing insights into user interactions and driving growth through increased conversion rates. Users can set up this connector using provided documentation and an entity relationship diagram (ERD) of the schema. Centralizing data from multiple sources enhances metrics for more powerful analysis.

Business Intelligence (BI) platforms such as Looker, Periscope Data, and Sisense offer robust functionality to explore, visualize, and analyze massive amounts of complex data; generate real-time reports; and answer pressing business questions. BI tools can handle large data volumes by using a collection of servers working in tandem, provide consistently up-to-date reports with real-time results, and give businesses control over their data through strict governance. Fivetran enables seamless integration of various data sources into a warehouse for effective BI analysis.

Fivetran offers automated, fully managed data pipelines and centralized data that can benefit businesses by reducing costs, saving time, increasing revenue, and building a data-literate company. The benefits include timelier and more accurate reporting, allowing analysts to dedicate their time to generating insights instead of manual data extraction. Automated, fully managed pipelines save developers time and minimize downtime. Fivetran's case studies showcase significant business successes, such as increased ROI, faster analysis completion times, and improved email conversion rates and transactions.

Fivetran has developed a new kind of data connector designed for the cloud era, exploiting advances in SaaS and cloud warehouses. The company recognized that traditional data replication technology was not built for the cloud and aimed to change this by building standardized connectors and incorporating key automations. This approach, known as ELT (extract, load, transform), leaves the transformation step to the discretion of the data analyst. Fivetran's engineering culture emphasizes simplifying ETL through standardization and automation, removing the engineering burden from data teams. The company's connectors are designed to detect and adjust to source changes, ensuring a consistent data flow for agile analytics. Deploying these data connectors can enable organizations to democratize data insights across various departments, leading to more informed decision-making.

Fivetran's approach to data management involves using standardized schemas across their data connectors to ensure uniformity in data representation for their customers, which is complemented by partnering with business intelligence platforms like Looker. Looker utilizes a proprietary language called LookML, which acts as a data modeling tool by simplifying SQL operations and structuring code through abstraction. Projects within LookML consist of models, views, and dashboards, facilitating collaborative work with version control via Git repositories. An example of this integration is the visualization of gross revenues from Salesforce data using LookML, which demonstrates the plug-and-play capabilities of Looker Blocks with Fivetran's standardized data schemas. This synergy aids organizations in efficiently solving analytics challenges without reinventing solutions, thus elevating the accessibility and baseline of data analysis for Fivetran's users.