March 2021 Summaries

This guide introduces data integration as a crucial step towards making data-driven decisions within an organization. It covers various aspects of analytics, including ad hoc reporting, business intelligence, data as a product, and artificial intelligence/machine learning. The text also discusses big data and its importance in analytics. Furthermore, it delves into the concepts of ETL (Extract-Transform-Load) and ELT (Extract-Load-Transform), explaining their differences and when each might be preferable. It also provides insights on building a modern data stack and the tradeoffs between building and buying your own data integration solution. Lastly, it offers practical steps to get started with data integration, such as assessing needs, migrating or starting fresh, evaluating elements of the modern data stack, calculating total cost of ownership and ROI, establishing success criteria, and setting up a proof of concept.

The article discusses the importance of maintaining data integrity in business operations, as bugs and errors can lead to significant problems if not addressed promptly. It highlights five strategies employed by Fivetran to ensure error-free data pipelines: building a solid foundation through rigorous testing and understanding of source systems; prioritizing bug resolution within 21 days on average; secure debugging with customer approval; providing public uptime calendars and incident reporting for transparency; and continuously improving reliability, performance, and security behind the scenes. The author encourages readers to test Fivetran's system through a 14-day free trial.

Fivetran has introduced a new code-free integration with Sage Intacct, allowing users to automatically load historical and updated accounting and finance data from the platform into their centralized warehouse for refined spend strategy. The benefits include easy access to complete financial data within analytics destinations, keeping destination data up-to-date, and enabling customers to merge Sage Intacct data with other sources for additional analysis. This integration is a valuable addition to the Sage Intacct Marketplace, providing business analysts with quicker access to their accounting and finance data.

In celebration of Women's History Month, Fivetran is featuring interviews with four outstanding women within the company. Today's spotlight is on Director of Product Management Alexa Maturana-Lowe. She shares her experience working with a manager who taught her to take up space and be confident in her worth. Her advice to her younger self is to spend more time exploring options and focusing on emotional growth. She admires Alexandria Ocasio-Cortez for her ability to speak powerfully, vulnerably, and publicly. If not in tech, Alexa would work at a plant nursery called Flowerland, as she loves plants and gardening. To ensure professional growth, she focuses on emotional development through reading books and working with a professional coach.

Pipedrive is a popular CRM platform for tracking sales activity, offering insights to identify successful deal cycles. Fivetran's latest connector allows users to centralize their Pipedrive data in their analytics destination, enhancing reporting capabilities and enabling better understanding of deal history, activities, and touchpoints. By connecting all data sources from Fivetran's directory of 150+ connectors, organizations can obtain more powerful and robust metrics for improved sales organization.

Rent the Runway's (RTR) data engineering team has evolved its role from handling ETL tasks to focusing on more strategic initiatives. The adoption of automated cloud-based tools has freed up time for the team, allowing them to develop generalized frameworks and set up a self-service data model. As reliable data ingestion becomes standard, RTR's data engineering team aims to become strategic advisors by understanding business context and guiding clients through cost and resource tradeoffs of different solutions. The team is also working on instilling trust in the data by ensuring accurate underlying information.

Adobe Analytics Data Feed integration by Fivetran allows users to generate website engagement insights using first-party data, which is crucial in a cookie-less marketing ecosystem. The integration provides more granular and customizable raw data compared to standard data pulls from Adobe Analytics. It also enables quicker transformation of data into insights with its hourly and daily data retrieval times. Users can consolidate and transform their data by mixing Adobe Analytics metrics with other data sources, allowing them to see the impact of website engagement on high-value outcomes like revenue and purchase price. To get started, users need an Adobe Analytics Data Feed account and can set up the connector here. For more technical information about Fivetran's Adobe Analytics Data Feed connector, visit their documentation.

Database replication for analytics involves numerous technical challenges such as incremental updates, idempotence, schema changes, history mode, data type mapping, replication and sync monitoring, optimization, security, support and assistance, and scalability. Addressing these issues is crucial to ensure efficient data operations, accurate insights, and minimized risks of data integrity problems or poor performance. Automated data integration solutions like Fivetran can help simplify the process by reducing human intervention and providing off-the-shelf support for various data sources and destinations.

Fivetran is a tool that helps companies gain visibility into their AWS environment by integrating key new services such as CloudTrail and Service Manager Inventory. These integrations enable businesses to centralize data from AWS CloudTrail, enhancing operational analytics with historical anomaly data for proactive future tracking. Additionally, Fivetran allows users to broaden their analytics programs by understanding how to structure team permissions through AWS IAM and identifying efficiency gaps in current infrastructure through a consolidated view of services. Companies can test every integration first to understand the best way to utilize this data in their operational analytics.

Companies are increasingly investing in real-time customer analytics to maintain their competitive advantage. Heap's product analytics solution helps capture simple events without repetitive pixel setup, and its integration with centralized analytics destinations can create a holistic analytics program. This includes understanding user actions for churn and retention, creating customer flows for product roadmap planning, and mapping user actions from trials to sales cycles for increased revenue. Fivetran offers a 14-day free trial of their Heap connector for customers.

In celebration of Women's History Month, Fivetran is featuring interviews with four outstanding women employees. Today's spotlight is on Dublin-based EMEA Marketing Director Ciara Roca. She shares insights about her career growth and the people who have positively impacted her journey. Ciara advises her younger self to embrace change and new experiences, build confidence, and seek support from a friend or mentor. To maintain professional growth, she participates in a mentoring group called Common Purpose, which brings together senior leaders from various sectors. She admires New Zealand Prime Minister Jacinda Arden for her empathetic leadership style and ability to balance work and family life. If not in tech, Ciara would be designing album covers for big bands. Fivetran is currently hiring across multiple teams and regions.

Fivetran has introduced new integrations with customer analytics tools Delighted and SurveyMonkey to improve support for customer experience tracking. These integrations allow users to automatically transfer their customer satisfaction data into a centralized data warehouse, enabling analysts to monitor various metrics such as Net Promoter Scores, customer loyalty, product ratings, employee engagement, and market research. The setup process is quick and straightforward using account credentials, with additional technical information available in Fivetran's documentation for both Delighted and SurveyMonkey connectors.

Strava, a social network for athletes, has seen significant growth in recent years and consistently seeks ways to improve business insights and accelerate data-driven decision-making. By embracing a modern data stack and educating internal stakeholders on the impact of data analytics, Strava's data team has been able to exceed expectations and foster Strava's data culture. The company's Data Engineering Lead Daniel Huang shared their process for scaling and meeting the growing company's data needs through automation and reimagining a new data culture. Key steps include realizing when change is needed, determining bottlenecks, and reaching scale via a modern data stack. Strava implemented a cloud data stack built on Snowflake as a data warehouse, Tableau as a BI tool, and Fivetran as its data pipeline provider to automatically land data in Snowflake from third-party vendors.

Process isolation is a crucial aspect of software engineering that ensures data pipelines remain secure and reliable by partitioning memory space and computational resources used in each process. This prevents interference between different applications or instances of the same application, guaranteeing that one process cannot write to or read from another. In the context of data pipelines and integration, process isolation means that every data connector for every customer is separated at a low level, ensuring utmost security and reliability. Process isolation guarantees security in two ways: by preventing data mix-ups between different customers using the same machine and by offering security from the standpoint of data governance. It also promotes reliability by isolating crashes resulting from bugs or memory shortages to a single instance, without affecting others. However, challenges posed by process isolation include determining the correct allocation of memory due to unpredictable data sync sizes and the need for wasteful over-allocation of memory to ensure that syncs virtually always succeed.

Charles Wang discusses the importance of using database connectors to uncover insights about operations in an organization. Production and operational systems that sit on top of databases continuously stream transactional data from various sources, including sensors, digital footprints, and ERP platforms. Metrics and KPIs can be tracked using data from these databases, falling into categories such as operations, product, and marketing. However, production and operational databases are not optimized for analytics queries, which require a fundamentally different approach to OLTP (online transactional processing). To efficiently analyze this data without interfering with operations, it must be moved from the databases to a data warehouse using data integration solutions like Fivetran.

Cloud-based data warehouses offer unlimited scalability and combine the best features of traditional data warehouses and data lakes. They provide top-notch support for structured, relational data and semi-structured data like JSON and XML. Combining a data lake and data warehouse into a single system reduces complexity, latency, and storage costs while improving data governance and searchability. Modernizing your data warehouse by incorporating the key features of data lakes can result in a simpler system with lower end-to-end latency for less money.

In celebration of Women's History Month, Fivetran is featuring interviews with four outstanding women employees. Today's spotlight is on Senior Analytics Technical Product Manager Veronica Zhai. She shares that her mother has had a significant positive impact on her career by instilling the values of excellence and independence. Her advice to her younger self includes having faith, working hard, persisting through challenges, and treating obstacles as learning opportunities. To sustain professional growth, she reads extensively, networks with other data professionals, and participates in industry conferences. She draws inspiration from a friend who started a non-profit organization offering free STEM tutoring to low-income students. If not in tech, Zhai sees herself becoming an entrepreneur or investing in minority-led businesses.

Zendesk Sunshine is a modern CRM platform that centralizes customer data, offering a comprehensive view of customers. By integrating this information with other data sources, organizations can improve their customer experience by tracking communication, leads, and sales activities. Fivetran provides reliable data management solutions to help businesses build an experience based on customer feedback. To get started, set up the Zendesk Sunshine connector and connect all relevant data sources for more powerful insights.

Operational analytics is an emerging method that delivers real-time data directly to teams and stakeholders within a company for improved operational efficiency. It differs from traditional analytics by automating the delivery of insights and data models, rather than relying on dashboards or reports. This approach makes it possible for companies to use their data warehouse as a new type of Customer Data Platform, enabling personalized experiences for customers. The rise of operational analytics is expected to create new roles and opportunities within data teams, such as the role of an analytics engineer who focuses on building pipelines that deliver real-time data automatically. With operational analytics, data becomes embedded in every department's operations, making it a vital undercurrent of insights facilitating success across the organization.

A global survey sponsored by Fivetran and conducted by Dimensional Research reveals that data engineers are struggling with outdated processes, hindering their ability to deliver maximum business value. The majority of data engineers lack the time and technological resources needed for centralizing valuable organizational data, transforming it into useful metrics, and making it accessible to various stakeholders. Two main issues contributing to this problem are data integration challenges and inefficient ETL (Extract, Transform, Load) processes. Automated data integration technology can address these core data engineering challenges by providing prebuilt pipelines that launch on demand, maintenance-free data integration, and accelerated data transformation and modeling. This allows data engineers to focus more on strategic projects such as managing and optimizing core data infrastructure, building custom ingestion pipelines, supporting data team resources with design and performance optimization, and building non-SQL transformation pipelines.

Fivetran is a data integration platform that helps businesses centralize their database data for analysis and insights. It offers efficient solutions to manage transactional databases, saving time and resources for companies across various industries. Businesses using Fivetran have reported improved customer loyalty, better understanding of Customer Lifetime Value (LTV), and effective allocation of marketing budgets. The platform's database connectors create reliable pipelines from transactional databases to analytics destinations, enabling up-to-date data usage for driving insights across all departments.

In celebration of International Women's Day and Women's History Month, Fivetran is spotlighting its female team members who are natural leaders contributing to the company's growth. The first in this series is Kiran Sharma, Lead Customer Success Engineer based in Bengaluru, India. She shares her inspiration from her mother, a state police officer who balanced work and family while raising five daughters. Kiran advises her younger self to speak up more often and not be afraid of being different. To sustain professional growth, she divides her attention into multiple areas, stays updated with global support developments, and learns from others' experiences. She draws inspiration from Whitney Herd, the CEO of Bumble, for her ability to drive change and take risks. If Kiran were not in tech, she would be a yoga teacher as it provides emotional and physical well-being.

As marketing evolves into a more data-driven discipline, attribution modeling has emerged as a key technique for understanding the effectiveness of various marketing channels in influencing customer conversions. Attribution modeling assigns credit to different touchpoints in the sales funnel, allowing marketers to optimize their strategies by identifying which channels drive the most conversions and improve return-on-investment. Various models exist, including first-click, last-click, linear, last non-direct click, time decay, and position-based, each offering unique insights into customer journeys. Marketers can implement these models using tools like Python, Google Analytics, or third-party solutions like Bizible, with each option offering varying degrees of complexity and customization. The ongoing shift towards digital analytics in marketing highlights the growing importance of data science in crafting more precise and effective marketing strategies.

Debezium is an open-source software solution for reliable data integration, which records transactions on databases and forwards them to Kafka clusters for downstream forwarding. It requires technical teams to implement new technologies and understand how to leverage Kafka effectively. Debezium with Kafka has challenges such as understanding transaction application to sink/target databases and handling duplicate records after connection failures. Debezium also doesn't handle schema drift or database migrations, requiring additional solutions for these tasks. Fivetran is an automated data integration tool that provides zero-configuration connectors, continuous synchronization of data from source to warehouse, and automation of in-warehouse transformations. It offers native support for various databases and warehouses, drastically reducing setup time and handling unexpected changes automatically.

Fivetran has released a new dbt package that consolidates ad spend, clicks, and impressions data from various advertising platforms such as Facebook, Google, Twitter, LinkedIn, Pinterest, and Bing. This unified view of advertising efforts can help businesses understand their marketing mix, identify what's working, and optimize their ad spend. The Modern Data Stack adoption helps get data out of platforms and into cloud data warehouses automatically, making it easier to analyze and gain insights from the aggregated data.

In today's fast-paced business environment, enterprises require instant access to data-driven insights for strategic decision making. However, legacy analytics processes are unable to deliver results quickly enough due to their reliance on manual coding and SQL-based ETL design and maintenance. This delays the desired BI outcomes and compromises business agility. Modern data stacks, centered around cloud data warehouses or lakes, offer advanced analytics capabilities and automate data engineering tasks, allowing businesses to respond more quickly to market changes and provide valuable insights for decision-makers.

The article discusses key considerations when choosing between Qlik Replicate (formerly Attunity) and Fivetran for cloud migration and ongoing replication of legacy ETL database software. It highlights that while both tools offer Change Data Capture support, Fivetran's methodology minimizes overhead on the destination environment. Additionally, Fivetran is a cloud-native SaaS application requiring no installation, whereas Attunity has prerequisites for setup and server monitoring. Furthermore, Fivetran supports more data sources than Qlik Replicate and allows users to freely choose their best-of-breed tools due to its vendor ecosystem independence.