March 2022 Summaries

A new report reveals that data engineers spend nearly half their time maintaining data pipelines, costing an average of $520,000 per year. This manual process is inefficient and consumes valuable engineering resources. By automating the data pipeline through managed data integration services, companies can save time and focus on more productive tasks such as building new models and analytics, democratizing access to data, developing internal apps, and continuously seeking opportunities for automation. Automating data integration not only optimizes modern data pipelines but also fosters a data-driven culture within organizations.

Extract, Load and Transform (ELT) is a data integration process that moves raw data directly from source to destination for analysts to transform as needed. It differs from ETL (Extract, Transform and Load), which has been the norm in data integration. ELT extracts and loads raw data with no assumptions about how it will be used, while ETL transforms data before loading it, limiting the volume of data that is warehoused to preserve resources. The modern cloud environment enables ELT by reducing storage, computation, and bandwidth costs. Fully managed ELT solutions automate pipeline maintenance tasks, freeing up data teams' time for analysis and insights generation.

Fivetran has introduced a solution for syncing Salesforce formula field values to data warehouses without manual translation from SOQL to Standard SQL. The feature includes the Fivetran_formula table and the Salesforce Formula Utils transformations package, which automatically maintain custom formulas and generate accurate representations of formula field values as views in the warehouse. This enables data analysts to access reliable and up-to-date Formula Field data for analysis alongside other sources. To get started, users need to sync their Salesforce data with Fivetran and use dbt Core for data modeling.

Data governance is essential for organizations to protect sensitive data, adhere to privacy regulations, improve data usage, analysis, and sharing across the company, increasing accountability as well as improving results. Effective data governance can add business value beyond compliance by improving discovery with metadata management, fueling effective advanced analytics, and providing a competitive advantage. Companies that fail to recognize the importance and value of governing data risk losing out to competitors.

Michel's article explores three different methods for syncing data from Snowflake to Google Ads, emphasizing the importance of fresh and accurate data for effective digital marketing. The methods include a manual CSV upload, a semi-automated process using Snowflake and Google Ads APIs, and a fully automated sync through Fivetran Activations. Each method allows organizations to leverage customer data for targeted advertising by creating lookalike audiences, exclusion lists, and direct targeting lists. The manual approach involves formatting data according to Google Ads requirements and uploading it directly, whereas the semi-automated method requires setting up API access and using programming languages like Python to facilitate data transfer. The fully automated solution with Fivetran Activations simplifies the process by automating data syncs, reducing the need for manual intervention and error handling, making it a more efficient choice for frequent data updates.

Google Cloud Cortex Framework, a set of reference architectures, packaged services, deployment templates, and accelerators for AI/ML, has been launched to enable organizations to leverage the full benefits of cloud innovation. Fivetran's HVR technology for the enterprise is a foundational partner with Google Cloud Cortex Framework. The partnership helps enterprises manage complex, high volume data needs while simplifying the process of SAP data replication into BigQuery. By using HVR to move SAP data into BigQuery and leveraging Cortex Framework, organizations can achieve broader insights for their business and benefit from infinite scalability and a low-risk, pay-only-for-what-you-use model offered by Google Cloud.

A data pipeline is a series of actions that combine data from multiple sources for analysis or visualization. It consists of three components: a source, a data transformation step, and a destination. Data pipelines enable businesses to centralize data from disparate sources into one place for analysis, ensuring consistent data quality and improving business intelligence. The modern data stack includes an automated data pipeline tool, a cloud data destination, a post-load transformation tool, and a business intelligence engine. Automated data connectors are the most effective way to reduce the programmer burden and enable data analysts and data scientists. Fivetran offers prebuilt and preconfigured data connectors that support over 150 data sources and manage normalization for ready-to-query data assets.

The article discusses a solution for securely onboarding customer data without handling their login credentials, which can introduce security issues and be cumbersome to manage. It introduces the concept of Connect Cards, setup forms by Fivetran that allow customers to authenticate connections between their sources and Fivetran. The article then provides a step-by-step guide on how to generate a link to a Connect Card using Python. This involves setting up a destination in your Fivetran account, gathering authentication credentials, building a Python file, creating a payload, generating the connector ID and token, and finally generating the link to the Connect Card. The generated link can be sent to customers via email or other communication mediums for them to authenticate the connection on their own without sharing sensitive information.

In this article, the author discusses her passion for increasing female participation in technology and emphasizes that there are many roles within tech companies that do not require technical skills. She shares her personal experiences of working in senior positions at successful tech companies and offers advice to women considering a career in technology or business. The author suggests choosing a space with growth potential, such as data management, and carefully selecting an employer based on their commitment to diversity and inclusion.

The modern data stack (MDS) is revolutionizing how businesses utilize ever-expanding data streams, making access to data as simple and reliable as electricity. It removes technical barriers, allowing individuals without in-depth technical knowledge to readily access data across various teams. Both SMBs and large enterprises can take advantage of the MDS, creating new opportunities, uncovering inefficiencies, and boosting productivity. Collaboration among industry leaders is essential for ensuring everyone benefits from this rapidly developing technology. The future of the modern data stack involves consolidation, greater unification, and working together on tightly integrated ecosystems.

HR analytics is a multidisciplinary approach to collecting, transforming, and normalizing HR data for insights that inform business decisions. It encompasses various data collection and analysis practices such as time to hire, employee turnover reasons, and engagement with upskilling programs. To produce the best results, HR analytics depends on diverse data sources from cloud-based human capital management systems or human resources information systems like Workday, ADP, BambooHR, Gusto, SAP SuccessFactors, Qualtrics EmployeeXM, and Lattice. Key applications of HR analytics include time to hire analysis, comparative compensation ratio analysis, PTO, overtime, and absenteeism analysis, feedback management, performance management, DEI metrics, and compliance monitoring. Integrating data from various systems into a central data warehouse can provide a holistic view of how HR impacts the business and improve overall organizational performance.

Observability is crucial for building resilient data systems, and the implementation of a hub-and-spoke model with reverse ETL, as shown by Wistia's experience with Fivetran Activations, can significantly enhance data pipeline efficiency. Wistia faced several challenges, including complex updates, language barriers, data sequencing requirements, lack of data visibility, and excessive alerts, which were addressed by centralizing data definitions and using dbt models with Fivetran Activations. This setup enabled faster updates, improved visibility, and reduced alert fatigue by ensuring that data from various sources like Salesforce, HubSpot, and Stripe is consistent and easily accessible. The new system allows for SQL-based changes, spreading the workload across a broader team familiar with SQL and Python, and ensures that data errors are promptly identified and resolved without unnecessary alerts. Wistia's success with this model highlights the potential for other companies to streamline their data operations and free up resources by reducing reliance on custom integrations, with future plans to apply the same model to their marketing automation platform, HubSpot, for consistent data handling.

The article discusses the importance of customer success strategy in a company's overall plan, emphasizing on the role of data in enhancing this strategy. It highlights three key touchpoints in a customer's journey - sales, support and success. Customer success is seen as a proactive function that monitors customers' interactions with products or services to guide them towards a more valuable experience. The article also presents three best practices for implementing an effective customer success approach: being selective and targeted, connecting the data across the organization, and putting the customer first. It concludes by suggesting various metrics that can be tracked in a customer success dashboard to anticipate customers' needs effectively.

To move data from BigQuery to Salesforce, three methods are outlined, each catering to different needs and skill levels. Data Loader is suitable for medium-sized, one-off syncs, allowing up to 5 million records, and involves exporting data from BigQuery via the GCP console and then importing it into Salesforce. The Data Import Wizard is ideal for smaller syncs, handling up to 50,000 records, and involves exporting data using the bq command-line tool. For continuous and automated syncs, Reverse ETL with Fivetran Activations is recommended, enabling the synchronization of both small and large volumes of data, using built-in SQL or no-code features, and offering automatic scheduling. Each method provides a specific approach to exporting data from BigQuery and importing it into Salesforce, ensuring flexibility and scalability according to the organization's needs.

In a company's data infrastructure, integrating disparate data sources into a central repository like Databricks is common, but moving this data to operational tools such as Salesforce can be challenging due to the lack of native sync integration. The blog details four methods for syncing data from Databricks to Salesforce, emphasizing the benefits of such integration for various teams like sales and marketing, who can leverage rich first-party data for enhanced performance. Exporting data involves using Databricks Notebooks, CLI, JSpark, or a more streamlined approach through reverse ETL using tools like Fivetran Activations, which bypasses manual CSV exports and allows seamless data synchronization from Databricks to multiple applications. The reverse ETL method offers a more efficient alternative by enabling scheduled syncs, ensuring Salesforce always has the most current data, supported by comprehensive documentation and customer support. Readers are encouraged to select the method that best suits their expertise and organizational needs.

Reliable data replication in the face of schema drift involves maintaining the integrity and continuity of data pipelines despite changes in data models, such as column, table, and data type alterations. Two primary methods are discussed: net-additive data integration, which retains both old and new schema elements to prevent data loss, and live updating, which directly mirrors source schema changes at the destination. Both methods have limitations, especially when data sources lack changelogs to track schema changes accurately. To address the challenge of tracking changes over time, history mode is introduced, which preserves all versions of row values with timestamps, though it may be resource-intensive. Additionally, adapting to data type changes involves selecting a supertype that can accommodate old and new values, ensuring seamless data transitions. Overall, reliable data replication demands more than simple data copying, requiring strategies to handle schema drift and historical data analysis, with tools like Fivetran offering solutions to these complex challenges.

Conversion tracking is crucial for digital advertising as it captures website interactions and powers ad reporting and follow-up marketing actions. Traditional conversion tracking relied on client-side web pixels, which have become less reliable due to privacy laws and technologies. In response, major social media platforms are introducing conversions APIs that allow server-side sharing of events. These APIs offer more powerful and flexible conversion tracking but require time-consuming engineering tasks for implementation. To improve data quality and ad performance with minimal development effort, centralizing all business data in a data warehouse using ELT is recommended. Automated data pipelines like Fivetran can move conversion events to the data warehouse, while reverse ETL services like Hightouch can send these events to conversions APIs. This approach minimizes marketing downtime and provides an immediate return on investment.