January 2026 Summaries

Rudger de Groot from Mintminds detailed how The Social Hub significantly reduced its experimentation costs by implementing GrowthBook with BigQuery optimizations from GA4Dataform by Superform Labs. By optimizing the cost structure, The Social Hub was able to conduct more experiments at a lower incremental cost, achieving an 81.8% reduction in BigQuery expenses while improving data refresh speeds and monitoring capabilities. GrowthBook's pricing model, which offers decreasing per-experiment costs with increased testing, along with the use of GA4Dataform's flattened datasets, enabled a dramatic decrease in data processing requirements and costs. This approach allowed for more frequent experiment updates and efficient use of BigQuery resources, resulting in substantial savings compared to traditional platforms like Convert.com Pro and VWO Pro. The collaboration between Mintminds and The Social Hub highlights the potential for achieving high-performance, cost-effective experimentation at a fraction of the price typically associated with enterprise solutions.

Generative AI has introduced a shift from deterministic to probabilistic engineering, challenging traditional software development paradigms by producing variable outputs and necessitating new evaluation methods. AI evaluations (Evals) check for a model's competence, while A/B testing assesses its value and impact on users, highlighting the need for both in AI product development. Vibe checking, a manual inspection method, is insufficient for scaling probabilistic systems due to its subjective nature. The industry has moved towards systematic AI Evals to quantitatively assess AI applications, yet these evaluations only measure the capability, not the user value, necessitating A/B testing to ascertain business impact such as retention and conversion. To optimize AI deployment, a staged pipeline involving offline evals, shadow mode, feature flags, and full A/B testing is recommended to filter risks and ensure both the competence and value of AI models before they reach end-users.

Optimizing for short-term A/B test wins using dark patterns can harm user trust and degrade product quality, as these tactics prioritize immediate business metrics over genuine user value and long-term satisfaction. Dark patterns, such as artificially degrading experiences, obscuring choices, or manipulating emotions, can lead to short-term gains but pose several long-term risks, including reputational damage, regulatory scrutiny, internal dissatisfaction, and competitive disadvantages. Although ethical guidelines and committees can help mitigate these risks, the most effective solution is to adopt metrics that reflect long-term outcomes and prioritize user trust and product enhancement. By focusing on retention, repeat usage, complaint rates, and brand sentiment, organizations can ensure their experimentation efforts align with creating real and lasting value, rather than exploiting users for temporary gains.