Balancing scale, cost, and performance in experimentation systems

A/B testing, while straightforward to initiate, becomes challenging to scale effectively without a robust data platform due to rising costs and potential system errors. The paper discusses designing an elastic and efficient experimentation system (EEES) that includes strategies for cost reduction, such as using big data technologies like Databricks, Snowflake, and Spark, and implementing an observability dashboard on BigQuery to identify bottlenecks. The design emphasizes separating metric definitions from logging to maintain data integrity, with pipelines structured to process raw data into actionable metrics. Transitioning from Databricks to Google BigQuery, and later incorporating Apache Iceberg with Spark, demonstrated significant cost savings and performance improvements, highlighting the importance of flexibility and adaptability in system architecture. The insights shared aim to help others build scalable and efficient A/B testing systems while avoiding common pitfalls.