October 2020 Summaries

Data enrichment involves enhancing customer data with additional context, but integrating it into existing Extract, Load, Transform (ELT) pipelines poses challenges such as maintaining data freshness, consistency, and managing resource usage. At Fivetran Activations, this process is optimized using the Clearbit API, Snowflake External Functions, and dbt to enrich large datasets efficiently. The approach distinguishes between enriching data via third-party datasets, which can be joined within a data warehouse, and APIs that provide data on a request basis, posing more challenges. Snowflake's external functions enable calling external APIs, like Clearbit's, within SQL queries, which are then managed through AWS infrastructure. This setup allows for caching to save API quotas and improve performance, with dbt managing incremental updates to maintain enriched data. Implementing this in production can be complex, requiring careful consideration of system components and potential pitfalls such as creating cycles in dbt's dependency graph, which are circumvented using source references. This method ultimately streamlines incorporating API data into declarative ELT workflows, presenting a promising alternative to traditional enrichment jobs, though it necessitates a good understanding of the involved technologies.

The text discusses the concept of Product Qualified Leads (PQLs) as a superior sales strategy in the SaaS industry, which leverages product usage data to convert freemium or trial users into paying customers. PQLs are identified based on specific user interactions with the product, such as sending a certain number of messages on Slack, indicating a higher likelihood of conversion compared to traditional Marketing Qualified Leads (MQLs) or Sales Qualified Leads (SQLs). This approach allows companies to tailor their sales processes to actual user behaviors rather than assumptions made from website interactions or contact forms. The text suggests using logistic regression to create a PQL scoring model by identifying key metrics that correlate with conversion, and emphasizes the importance of having a unified data source to derive these insights. Companies like WorkOS and Bonjoro have successfully implemented PQL strategies, resulting in higher conversion rates and reduced customer churn. The text also outlines steps to operationalize PQLs, including data organization, scoring, routing leads to appropriate sales channels, and customizing sales engagement strategies. Additionally, Fivetran Activations is highlighted as a tool to synchronize data across platforms, facilitating the effective use of PQLs in business operations.

Building a data-driven culture is crucial for businesses, but it requires more than just hiring a data analyst. Companies often face challenges in getting started with data due to lack of clear data ownership, absence of data democratization, and unsuitable tools and technologies. To build a data-driven culture, companies should communicate the value of data early and often, use self-serve, automated tooling, ensure data observability and reliability, and scale their impact by building a community around data.

The article discusses the Modern Culture of Data framework developed by Slalom to enable organizations to shift from siloed and risk-averse practices to collaborative, experimental, and data-driven workflows. One dimension featured in this framework is Access and Transparency, which contains three competencies: Data Complexity, Architecture and Infrastructure, and Delivery and Consumption. The article explains each of these competencies and their opposite ends, providing insights into how organizations can improve their data access and transparency capabilities with the help of Fivetran and automated data integration.

The latest dbt (data build tool) package accelerates Mixpanel event analytics by cleaning and aggregating data into tables that categorize users, aggregate events over time periods, and group events by sessions. Additionally, a macro generates customer conversion funnels to understand the journeys leading to valuable conversions. While Mixpanel API offers Query, Export, and Data Pipelines APIs for data export, Fivetran's Mixpanel connector simplifies setup and syncing without requiring manual intervention or extensive API documentation reading.

The first-ever Modern Data Stack Conference will take place on October 21 and 22, featuring speakers from the fields of data analysis and engineering. The free online event is aimed at data analysts, scientists, engineers, and other professionals in the industry. Key topics to be covered include analytics engineering, data preparation, and trends in data infrastructure. Notable speakers include former Snowflake CEO Bob Muglia, Fishtown Analytics CEO Tristan Handy, and experts from companies such as Peloton, Fastly, and Fivetran. The conference will offer opportunities for participants to deepen their knowledge, sharpen their skills, and connect with peers in the data industry.

Fivetran and Segment are two distinct platforms in the data integration space. While Segment is a customer data platform that focuses on aggregating customer events for efficient targeting, Fivetran is an automated data integration tool designed to provide consistent, reliable access to data for analytics organizations. Key differences between the two include their primary focus areas, schemas and APIs support, historical data ingestion, entity relationship diagrams, and marketing sources connectivity. Companies often use both platforms together, with Segment for initial event aggregation and user profile creation, followed by Fivetran to connect marketing sources fed by Segment audiences to their data warehouse.

Automated data pipelines can improve the efficiency of data analysis by allowing analysts to access the data they need without relying on development teams for approval processes. These pipelines offer prebuilt connectors, automatic data updates, and automatic schema migration, which help reduce the burden on engineering teams. By automating data integration, businesses can unite analytics and development teams, enabling faster delivery of insights based on real-time data.

Fivetran CEO George Fraser and Fishtown Analytics CEO Tristan Handy discussed the benefits of the Fivetran-dbt integration in a Slack conversation, emphasizing its value for data analytics engineering enthusiasts. Both companies believe that this integration offers an ideal toolset for modern analytics and look forward to collaborating on shaping the future of data movement and transformation. The full conversation can be read on the dbt blog.

Airbnb has become a leader in data-driven decision-making by developing a sophisticated data infrastructure and launching initiatives like Data University, which democratizes data access and education across the company. Founded in 2016, Data University was inspired by Google's internal training programs and aimed to provide all employees, regardless of role, with the skills to independently analyze data. This initiative was crucial in making the company more data-informed, with metrics showing significant increases in data platform engagement and profitability shortly after its launch. By prioritizing fundamental data education and equipping employees with necessary tools, Airbnb empowered its workforce to make autonomous, data-driven decisions, which in turn amplified the impact of its relatively small data science team. The program's success is evident through its expansion to include intensive training sessions and the Engineering Empowered Data Science program, further enhancing skills and collaboration. Airbnb's approach underscores the importance of training employees as "citizen data scientists" and highlights the organizational benefits of transparency and data proficiency in decision-making.

The first-ever Modern Data Stack Conference will take place on October 21 and 22, featuring speakers from the fields of data analysis and engineering. The free online event is aimed at data analysts, scientists, engineers, and other professionals. Day 1 focuses on business and analytics, with presentations by Fivetran co-founders George Fraser and Taylor Brown, Data Culture co-founders Leah Weiss and Gabi Steele, F5 Networks' Lance Hokanson, Dialpad's Jacob Bedard, Emery Sapp and Sons' Clayton Hicklin, data visualization expert Kate Strachnyi, Clari's Gary Sahota, and an analyst panel led by Interworks' Brian Bickell. The conference aims to deepen knowledge, sharpen skills, and connect participants with their peers.

Automated data integration can significantly improve machine learning efforts by reducing the time spent on data munging tasks such as obtaining, cleaning, and preparing data for analysis. By utilizing The Modern Data Stack, which consists of third-party ingestion, a cloud data warehouse/data lake, an in-warehouse data modeling layer, and a BI tool, data scientists can streamline their ML modeling workflow and spend more time on value-adding tasks. This approach also ensures that business concepts are pre-calculated and readily available for ML modeling, reducing the need for custom software and promoting consistency across the organization.

Enterprises are facing challenges in ensuring accurate and actionable data for critical business decisions due to complex integration processes, stale and unreliable data, and unmet needs. Automated ELT (extract, load, transform) is a modern solution that provides nearly instant access to data and allows teams to allocate resources more effectively. Fivetran automated data connectors support over 150 data sources and adapt to changes in schemas, ensuring analysis-ready data assets for enterprises. A webinar with industry analyst David Loshin further explores the benefits of automation in reducing operational risk and simplifying data integration management.

The architecture of automated data integration is based on several key concepts, including extracting, loading, and transforming data with minimal human intervention. Automated data integration is closely tied to the growth of cloud computing, which has led to cheaper storage and processing costs, enabling a profusion of cloud-based services. Fivetran's modern data stack includes components such as data sources, automated data pipelines, data warehouses or lakes, transformation layers, and business intelligence or data science platforms. Design considerations for an automated data pipeline include ELT vs. ETL, incremental updates, idempotence, normalized schemas, schema migrations, net-additive data integration, and data type detection and coercion. Other important factors to consider are reliability, scalability, and security.

Fivetran and Stitch are both automated ELT solutions, but they differ in their approaches to handling data integration. Fivetran provides entity relationship diagrams (ERDs) for table relations and supports post-load transformation options. In contrast, Stitch displays depth of data integrations in a tabular format and uses raw rows for pricing models. Fivetran uses active rows, which only count one "active" row regardless of record updates or deletes. Additionally, Fivetran has removed its minimum spend requirements and offers online purchasing through credit cards. Both platforms offer free trials to help customers make informed decisions.

Fivetran and Stitch are both automated ELT solutions that have been commonly evaluated by customers searching for cloud data integration tools. However, they differ in their approaches to handling automated ELT and pricing models. While Stitch uses raw rows for its pricing model, Fivetran uses active rows, which only counts one "active" row regardless of the number of updates or deletes. Additionally, Fivetran has removed its minimum spend requirements and offers online purchasing through credit cards. Both platforms provide some pre-load transformations, but only Fivetran provides post-load transformation options for additional modeling with off-the-shelf data models.

The Modern Data Stack Conference is a community-based event that brings together practitioners and leaders to discuss how modern data stacks can transform analytics programs. With keynote speakers from Fivetran, Andreessen Horowitz, Fishtown Analytics, Notable, and Snowflake, the conference will feature two half-days of learning on October 21st and 22nd, focusing on business/data analytics and data engineering respectively. The event aims to provide insights into cloud data platform strategies, SaaS BI solutions, building and maintaining data pipelines, and optimizing existing stacks. Registration is open for the free online conference at https://mdscon.fivetran.com/.