Salesforce to BigQuery ETL with Google Cloud

Over the summer of 2023, the Vantage Team implemented a data pipeline using Google Cloud to transfer Salesforce data into BigQuery, enabling daily snapshots of their sales pipeline for improved business insights. The project aimed to capture key sales metrics such as pipeline coverage, close rate, and pipeline creation by taking daily snapshots from Salesforce, which were then stored in BigQuery using Google Cloud Data Fusion (CDF). They employed CDAP, now known as Google Cloud Data Fusion, to manage the ETL process, utilizing the Salesforce Batch Multi Source plugin to create daily snapshot tables, which were later transformed into a clean table for analysis. The team faced challenges such as managing Salesforce authentication resets and optimizing cloud costs, which initially spiked due to non-stop running of the pipeline. By enabling autoscaling on CDF, they reduced costs significantly. The setup allowed for the agile addition of new Salesforce fields into BigQuery, demonstrating flexibility in schema management, and helped inform strategic decisions in sales and marketing by providing a clearer view of their sales pipeline dynamics.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.