How we do performance engineering at Fivetran

Fivetran describes how it substantially improved data-pipeline performance after its existing incremental optimization efforts failed to meet an enterprise target of moving 1 TB of data in two hours. Starting from benchmarks of 26 MB/s for Oracle HVA-to-Snowflake and 48 MB/s for Postgres-to-BigQuery, the company ultimately reached 139 MB/s and 259 MB/s respectively, while increasing core production throughput from 7 MB/s to 70 MB/s. Its approach centered on establishing serialized extract volume as a consistent throughput metric, instrumenting every pipeline phase, creating repeatable benchmarks with controlled environments, building visualization and profiling tools, and documenting both gains and regressions. A cross-functional temporary “Tiger Team” was given protected focus and authority across connector, core, destination, infrastructure, and quality-engineering boundaries, enabling system-wide changes such as parallel processing, multithreaded imports, serialization improvements, and connector rewrites. After meeting its goals through roughly 50 projects, Fivetran disbanded the temporary team and formed a permanent Platform Engineering performance group to maintain benchmarks, prevent regressions, improve efficiency and scalability, and support customer-specific performance needs.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.