April 2021 Summaries

Airtable offers the functionality of spreadsheets in a cloud-based environment, providing various visualization and organization options that can be tailored depending on departmental needs within an organization. The Airtable connector, developed by Fivetran, allows users to integrate their Airtable data with a centralized analytics destination, facilitating activities such as building product launch dashboards, orchestrating marketing campaigns, and streamlining financial monitoring. Users can set up this integration using an API key, which enables access to Airtable's API, including tables, metadata, and webhooks for incremental updates. Fivetran's partnership with Airtable enhances this integration by allowing access to additional fields and providing increased throughput for faster data syncing. For further technical details and to explore a wider range of metrics, users are encouraged to connect with other data sources through Fivetran's extensive directory of connectors.

The article discusses the importance of data integration in businesses and how it can help identify patterns, improve visibility into operations, and formulate new strategies. It explains that data integration requires a data pipeline, often referred to as an ELT (extract-load-transform) pipeline, which is more sustainable, flexible, and easier to automate than the traditional ETL (extract-transform-load) pipeline. The article also highlights the challenges of choosing the right tools and platforms for data pipelines, warehouses, and business intelligence, emphasizing that the most expensive cost is often not the product itself but the human cost of talent and labor. It suggests that organizations should prioritize solutions with ease-of-use, automation, and self-maintenance to save on labor costs. The article also recommends Fivetran as a data pipeline solution due to its wide range of supported source systems, simple interface, and automated tasks.

SAP Concur is a travel and expense management tool that helps companies track their financial health metrics. Fivetran has developed an integration for reliable, easy-to-use ELT of SAP Concur data into financial analytics platforms. The integration addresses technical challenges such as parsing API documentation, understanding system fields, and creating import strategies. Users can consolidate expense-related data from SAP Concur into their analytics destination to gain insights on spend tracking, ROI analysis, and lead generation for marketing events. By setting up the connector, teams can quickly access and analyze SAP Concur data to improve performance and troubleshoot failures.

Fivetran has announced a significant expansion of its managed service offering, Powered by Fivetran, which helps businesses build data-powered experiences on world-class infrastructure. Since its release in June 2020, the product has seen rapid adoption and was recently named 2021 Data Integration Innovation of the Year at the Data Breakthrough Awards. Key new enhancements include OAuth token support for connectors, connector metadata API endpoint, an API endpoint for columns, and a custom-branded OEM option. The expansion aims to better serve customers and save them valuable time to focus on delivering value to their clients.

Data orchestration is crucial for managing modern data pipelines, ensuring reliability and efficiency. Traditional ETL processes required custom-built transformations by engineering teams, while DIY scheduling methods did not scale well. Modern orchestration tools leverage programming languages like Python to design complex, robust, and dynamic systems. These tools follow the workflows-as-code paradigm and include products such as Luigi, Apache Airflow, Dagster, Prefect, AWS Step, Google Cloud Composer, Argo, and Tekton. The modern data stack consists of specialized tools that handle orchestration internally, reducing the need for separate orchestration tools in many cases. However, specific use cases may still require in-house orchestration systems, such as ETL processes with unique requirements or custom scripts for data products. Overall, data orchestration is a vital component of modern data management, but its role varies depending on the business's needs and the capabilities of their chosen tools.

A new Python package from Fivetran and Astronomer enables connector management in Apache Airflow. The Fivetran Provider allows users to start and monitor Fivetran data syncs directly within Airflow, combining the benefits of automated data ingestion from Fivetran and data pipeline orchestration from Airflow. This integration offers three main benefits: syncing syncs, triggering transformations, and managing DataOps. The package can be downloaded by running pip install airflow-provider-fivetran.

Fivetran, a data integration startup, has evolved from a five-engineer team to a 200-engineer unicorn with a focus on making business data as accessible as electricity. The company's growth and success can be attributed to its dedication to improving the reliability of its data pipeline through four distinct stages: crawl, walk, run, and fly. Key aspects of this evolution include monitoring and observability, incident management, handling source API failures, tracking changes on the source side, customer support, and building a modern data stack strategy with customers. Fivetran's commitment to reliability has enabled it to become an essential tool for businesses seeking to break down data silos and gain a holistic view of their organizations.

The article discusses the challenges faced by companies in integrating data from various external sources into a single platform for analysis and operational use. Traditionally, this involved custom code connections created by data engineers to connect data APIs, which was time-consuming and required ongoing maintenance. Automated data connectors are proposed as an alternative solution that can easily connect to a wide range of data sources with minimal configuration, coding, and user input. These automated connectors reduce the need for data engineers to develop custom code or infrastructure to manage complex API connectors, thereby saving development and maintenance time. The article also highlights the shift from ETL (extract-transform-load) to ELT (extract-load-transform), which allows faster access to data by analysts and data scientists. Automated connectors can help free up data engineers' time for higher-value projects while enabling quicker insights from raw data.

Fivetran has joined Asana Partners as a Technology Partner, enabling joint customers to enhance their analytics programs through an out-of-the-box integration. Asana helps teams manage and organize work, with over 93,000 organizations worldwide using the platform. The partnership ensures continuous updates to Fivetran's connector for driving outcomes in product and operational analytics. This collaboration aims to create integrated workflows that allow customers to achieve success, whether working remotely or in-person.

Fivetran and Supermetrics are two marketing data integration tools with different features and pricing models. While both offer a 14-day free trial, their license model differs; Supermetrics starts at $39/month and increases based on the number of connectors, accounts, and users, while Fivetran uses consumption-based pricing with tiered options for Starter, Standard Select, Standard, and Enterprise. Supermetrics is built for Excel and focuses on downloading and uploading CSV files from marketing channels, whereas Fivetran is designed for data warehouses and automates the process of ingesting data into cloud data warehouses. Fivetran has more connectors (180+) compared to Supermetrics' 65, and supports a wider range of data warehouses. Additionally, Fivetran provides pre-built query-ready schemas and flexible data sync times, while Supermetrics requires manual table-by-table loads. Both tools offer varying levels of support and documentation, with Fivetran offering more comprehensive resources and customer assistance. Ultimately, the choice between Fivetran and Supermetrics depends on a company's specific needs and priorities in marketing data integration.

Companies that operate B2B and B2C channels face complex management challenges, which can be solved by using a modern data stack. These companies often struggle to manage B2C processes and intelligence alongside increasingly important B2B processes and intelligence. A common set of questions arises for mixed B2C/B2B companies, including how to acquire and monitor individual vs. enterprise leads through the sales funnel, what are the strongest indicators of lead conversion, and how to develop internal processes that support both client types without increasing overhead. The solution is to migrate operational data from a company's business systems into a cloud data warehouse like BigQuery, Snowflake, Redshift, or Azure Synapse. This approach enables analytics professionals to iterate efficiently on their modeling and analysis while allowing the action to be taken by marketing, sales, and customer success teams. Automating and outsourcing the first two steps of the migration process (Extract and Load) allows data engineers and business intelligence analysts to concentrate on projects that drive value for business stakeholders rather than plumbing work.

Optoro is a technology company that helps manage returned items by finding them new homes, ultimately saving customers money and reducing waste. The company's platform generates large amounts of operational data as returned inventory is routed through various systems. To ensure the quality and reliability of this mission-critical data, Optoro implemented an automated data observability solution using Monte Carlo, a cloud-based tool that learns a company's unique data patterns over time and detects anomalies. By monitoring their entire data ecosystem, Optoro can quickly identify issues and maintain trust in their data among internal stakeholders and clients. The implementation of this modern data stack has saved about four hours of work per data engineer per week and improved overall confidence in the company's data integrity.

Data scientists can significantly enhance innovation and growth within businesses, but it is crucial to establish a robust foundation to maximize their value. Many companies rush to hire data scientists without adequate preparation, resulting in these professionals performing basic data integration and reporting tasks rather than utilizing their specialized skills. The readiness to hire data scientists can be evaluated through the "data hierarchy of needs," which begins with a solid enterprise data warehouse that consolidates all business data in a modern, cloud-based system. Following this, a clean and curated data view is essential to resolve inconsistencies and duplicate concepts across systems. The next level involves traditional business intelligence and analytics, which support daily decision-making through tools like SQL, Tableau, or Looker. Only when these foundational layers are in place can data scientists effectively apply advanced statistics and modeling, akin to star athletes on a well-supported sports team. Businesses should ensure these prerequisites are met to fully leverage the potential of data scientists.

Adopting a data-driven approach requires aligning people, processes and technologies. Automating data integration is crucial for centralizing data from multiple sources. Identifying a proof of concept (POC) with an influential stakeholder can drive interest in using data for decision-making. Ensuring insights lead to action involves making data visual and easy to understand. Providing a single source of truth for business users helps eliminate inconsistencies or inaccuracies in the data. Democratizing data by implementing data fluency training empowers employees to access and interrogate data independently, leading to more data-driven decisions across the organization.

Fivetran emphasizes its commitment to customer success, highlighting the importance of reliable support and technical expertise in managing data pipelines. The company provides dedicated teams for customer support, technical success, and customer success to ensure seamless data flow and address any issues promptly. This focus extends beyond the initial sales process, with Fivetran offering tailored data ingestion strategies and a 14-day free trial to prospective customers. The company boasts a 99.9% uptime and a high customer satisfaction score, underpinned by robust global teams working on maintaining and updating their products, including free dbt packages for connectors. Fivetran's infrastructure supports over 700,000 schema changes monthly, ensuring data pipelines remain operational even when source systems change, thereby empowering analytics teams to make informed decisions.

Fivetran has expanded its operations in the Asia-Pacific region with the opening of a new cloud region in Singapore and the formation of a local team. The expansion follows the launch of their APAC HQ in Sydney last year, as part of their planned rapid growth strategy for the region. The Singapore team will provide customer support across various areas including sales, marketing, partnerships, and customer service. This move enhances Fivetran's ability to offer 24/7 global customer support and improve compliance with local data protection laws such as the Singapore Personal Data Protection Act (PDPA).

Reverse ETL is an emerging solution in the modern data stack that enables businesses to utilize their analytics effectively by copying data from cloud data warehouses into operational systems of record, such as CRM tools and marketing platforms. This process completes the loop of data integration by making insights actionable for business teams. The core reason behind Reverse ETL is to build delightful, data-driven customer experiences by ensuring accurate data is available in downstream systems used by various teams to engage with customers. Operational analytics refers to feeding insights from analytics to business teams in their usual workflow so they can make more data-informed decisions. With the increasing amount of customer data being collected and the decreasing costs of storing data in cloud data warehouses, businesses already have what it takes to meet the expectations of demanding customers.

Fivetran has introduced a new Splunk integration that allows teams to monitor and investigate their infrastructure data more effectively. Splunk is a software platform designed for DevOps teams, enabling them to gather and manage operational intelligence by searching, monitoring, and analyzing machine-generated data from IT infrastructures. Fivetran's connector enables quick import of data from Splunk, allowing users to customize their data pipeline according to organizational goals. This integration can help companies improve developer efficiency, iterate faster, and gain valuable insights into infrastructure efficiency and security analytics. To get started with the new Splunk connector, visit Fivetran's documentation or set up the connector if you have a Splunk account.

The article discusses how Extract-Transform-Load (ETL), a traditional method of data integration, is being disrupted by the more modern and efficient approach called ELT (Extract-Load-Transform). ETL has become labor-intensive, brittle, and complex due to its reliance on custom software solutions designed and built by engineers. On the other hand, ELT decouples extraction and transformation, moving data into a data warehouse before applying business logic to transform it into data models for analytics. This approach is analyst-centric, allowing shorter turnaround times for all analytics projects and timelier delivery of insights. The benefits of ELT include automation, outsourcing, and the ability to substitute technology for labor, making it more accessible and practical for companies of all sizes. As ELT platforms continue to mature and the cost of cloud infrastructure declines, most firms will eventually find no advantage to using ETL over ELT.

Calix, a company that provides software platforms and services to communication service providers (CSPs), has centralized its data operations process to deliver more value faster. The Senior Director of Analytics and Decision Science at Calix, Pete Watridge, realized the need for simplification in the company's own data operations when he joined. By implementing a modern data stack using Snowflake, Fivetran, and dbt, Calix has seen benefits such as richer analyses and a single source of truth. The team is also focusing on business outcomes and reimagining the role of its analysts to act more like project managers with both technical ability and domain knowledge in their respective functional areas.