February 2022 Summaries

Fivetran, a company that provides automated data integration solutions, has experienced significant growth in India since launching operations there in January 2021. Within 11 months of launch, the company achieved an annual recurring revenue (ARR) of $1 million and saw its customer base more than double. Fivetran's success is attributed to its ability to provide powerful automated connectors that can centralize data from various sources, helping local businesses improve efficiency and realize cost savings. The company currently has 100 employees in India and plans to hire more sales, technical support, and customer success staff in the coming months. Fivetran also plans to bring a new AWS data center to India later this spring, further expanding its presence in the country.

This article discusses how Untitled Firm, a company that builds and manages modern data stacks, uses Powered by Fivetran to effortlessly connect to its customer's data and create powerful analytics applications. It highlights the challenges of connecting to external data sources and presents two possible solutions: asking customers for their credentials or allowing them to grant access themselves. Untitled Firm opts for the latter approach using Fivetran Connect Cards, which provides a secure front-end authentication experience for its customers. The back-end architecture involves aggregating raw data with dbt Core and applying custom models for visualization and presentation. This method automates the data onboarding process, enabling Untitled Firm to quickly scale its application and support growing customer needs.

Operational Analytics is an approach that enhances organizational data usage by integrating trusted and accessible data from warehouses directly into frontline tools like Salesforce, Hubspot, and Marketo, allowing teams to act on insights rather than just analyze them. This method addresses the limitations of traditional data tools that often fail to maintain data quality, freshness, and availability at scale. It leverages the modern data stack and the principles of DataOps to treat data as a product, facilitating continuous delivery and real-time insights to core applications. By bridging the gap between data and operational teams, Operational Analytics fosters better automation, cross-functional understanding, and efficient workflows, driving strategic data use across marketing, sales, and customer success teams. This approach aims to empower data teams to focus on strategic initiatives rather than cumbersome integration work, ultimately enabling businesses to deliver personalized, timely actions and insights, thereby enhancing customer relationships and operational efficiency.

Data transformation is crucial for converting raw data into analysis-ready states that can be used to generate meaningful insights. Fivetran Data Models are pre-built tools that help data analysts and engineers transform raw data from common sources into data models for reports and dashboards, saving significant time and effort. In this blog, the process of creating a running snapshot of a sales team's performance using the Fivetran data model for Salesforce is discussed. The Salesforce data model allows for reading from several raw tables such as account, user, opportunity, and user role. These are then transformed into finished data models like sales snapshot, owner performance, manager performance, and opportunity enhanced. The steps to produce the sales snapshot data model, which summarizes bookings, pipeline, losses, and wins in the last month and quarter, involve staging the data, determining whether each opportunity was created and/or closed in the specified timeframe, joining tables, assigning statuses to opportunities, creating fields for amounts and counts by quarter and month, and finally aggregating data from opportunity enhanced by status. The resulting data model provides metrics that can be used to create reports or dashboards summarizing a sales team's performance over the past month and quarter.

In 2022, the data analytics landscape is expected to undergo significant changes due to advancements in technology and evolving best practices. Fully managed ELT pipelines will offer increased automation and performance, allowing companies real-time access to all their data. DataOps processes are becoming more competitive as companies integrate these features into their products. Operational analytics will continue to improve internal processes and customer experiences by making data more accessible throughout organizations. BI tools will enhance collaboration between data teams and business units, facilitating seamless communication about analytics. The continued rise of automation through coding frameworks like Apache Spark, Beam, Flink, and PySpark will provide competitive advantages to companies that leverage these tools effectively.

The article discusses the need for an open-source library ecosystem in SQL, similar to what exists in languages like C++ and Java. It highlights that SQL lacks such a system due to different implementations providing incompatible extension mechanisms and the absence of a package manager. PostGIS is cited as a counterexample, having overcome these issues by targeting a single database management system and becoming popular enough for major vendors to pre-package it. The article suggests that building separate libraries for each SQL implementation could be feasible given the few popular systems in analytical databases. Additionally, dbt's package manager could potentially distribute open-source libraries of user-defined functions. This approach has already begun with Fivetran distributing over 50 data models to customers through dbt package hub.

Data marts are miniature, specialized data warehouses that provide access to specific lines of business or teams within an organization. They serve as single sources of truth and integrate easily with business intelligence platforms for report and dashboard creation. Unlike data warehouses, data marts do not include all organizational data but may consist of analysis-ready data models and raw data selections. Traditionally, data marts were physically separate from data warehouses, but the modern approach involves organizing them within a cloud-based data warehouse using logical data models like views and materialized views. This method enables granular access control, efficient management, and leverages the scalability of cloud technology.

Fivetran's fully managed data pipelines are now available in Microsoft Azure Marketplace, allowing customers to discover, transact, and deploy solutions optimized for Azure. This integration simplifies the purchasing process, helps companies maximize their Azure investment, and enables faster integration with Fivetran-supported data sources. Fivetran's automated data pipelines can extract and load data into cloud data warehouses like Azure Synapse, Azure Databricks, and Snowflake. Additionally, Fivetran Business Critical is available for enterprises with strict compliance requirements.

In 2021, Fivetran received 11 Best Places to Work awards, reflecting its strong company culture. The company launched the Fivetran Parents ERG to support parents and guardians, adding to existing groups for women+, the LBGTQIA community, black employees, and Latinx employees. Employees also engaged in charitable activities, raising thousands of dollars for various causes. Career development opportunities included an exchange program between U.S. and Amsterdam engineers, a research rotation program, and Camp Fivetran, a company retreat focused on relationship building. Despite the challenges of 2021, Fivetran's strong culture brought its workforce together, leading to significant product achievements and growth. The company is currently hiring for various positions across departments and regions.

Data lakes serve as central destinations for business data and offer users a platform to guide business decisions. Unlike data warehouses and data marts that require structured data, data lakes can accommodate large volumes of both raw, unstructured data and structured, relational data. They are popular for use cases such as storing huge volumes of data before modeling it and loading it to a data warehouse or serving as specialized destinations for specific AI/ML applications. However, without proper data governance, data lakes can become "murky" and difficult to navigate. New technologies like AWS Lake Formation and Databricks Data Lakehouse are combining characteristics of both data warehouses and data lakes to make data less murky. Single sources of truth such as data warehouses and data lakes will continue to form the foundation of modern data stacks, enabling analytics through data integration.

Data transformation is crucial to data modeling and enhancing an organization's data literacy. Businesses analyze data to gain insights about customers, discover new opportunities, and make informed decisions. To be useful, data must be organized and structured for easy interpretation by analysts. ELT software plays a vital role in this process by extracting data from various source systems, loading it into a repository, and transforming it for analysis purposes.

Data transformation is crucial to data modeling and enhancing an organization's data literacy. Businesses analyze data to gain insights about customers, discover new opportunities, and make informed decisions. To analyze data effectively, it must be organized and structured in a way that can be easily interpreted by analysts for creating dashboards, reports, and predictive models. Data is valuable only if it can be utilized effectively. Similar to how raw ore gains value after mining, refining, and shaping into something useful, data from business applications like Salesforce and Google Analytics needs extraction, loading into a repository, and transformation for deriving business intelligence. ELT software plays a vital role in this process by extracting data from various source systems, loading it into a data warehouse or lake, and transforming it into a suitable format for analysis.

Fivetran has announced updates to its pricing plans, aiming for increased transparency and accessibility. The company is making resyncs free for all connectors, phasing out credits in favor of a dollar-based commitment model, reducing costs for small teams with the new Standard Select plan, updating key features of its pricing plans, and adjusting its pricing curve to create a more linear relationship between spend and monthly active rows (MAR). These changes are expected to improve Fivetran's services and allow for the introduction of new feedback-driven features.