September 2026 Summaries

No summary generated yet.

Supercell conducts relatively few A/B tests despite serving 300 million monthly players, prioritizing high-impact, hypothesis-driven experiments that preserve its decentralized creative culture and focus on player retention rather than revenue. Its central experimentation team provides testing infrastructure and promotes rigor while allowing independent game teams to make their own decisions, using experiments to validate or challenge creative ideas rather than simply follow dashboard metrics. To maintain player trust, Supercell publicly announces tests, explains their purpose, and offers make-up events to ensure players who receive less favorable test experiences are treated fairly. AI tools are now enabling more employees to independently analyze experiments, expanding access to data but raising concerns about analytical quality and overconfident errors. Supercell and its experimentation partners view templates, guardrails, and standardized defaults as essential for embedding expert-level rigor into self-service AI analysis, reinforcing the broader principle that trustworthy experimentation depends on transparency, fairness, and reliable systems.

Scaling A/B testing requires more than increasing experiment volume; it depends on a company-wide operating model that preserves reliable measurement, shared learning, and quality controls across teams. The text identifies five core pillars: a scalable warehouse-native technical foundation with consistent metric definitions and A/A validation tests; rigorous prioritization frameworks that weigh expected impact against cost; an integrated experiment repository that records hypotheses, results, and reusable learnings; statistical safeguards such as pre-committed designs, automated health checks, multiple-testing corrections, variance reduction, and replication of surprising results; and self-service tools governed by required review workflows. Examples from organizations including DoorDash, Disney, Home Depot, Chess.com, Fyxer, the Philadelphia Inquirer, and Lingokids illustrate approaches to running experiments at different scales. It argues that programs should judge success not by test counts alone, but by validated product improvements, avoided harmful launches, and accumulated knowledge about users, while presenting GrowthBook as a platform intended to support these practices.