Why Twilio ships on signals instead of significance
Blog post from GrowthBook
Wanli Lau, now leading global R&D analytics at Twilio after eleven years at Expedia Group, discusses how experimentation principles change between high-volume consumer marketplaces and lower-volume B2B developer platforms. At Expedia, an account-signup homepage takeover initially increased conversions but later proved harmful to return visits and engagement, prompting the team to make guardrail metrics, customer journey mapping, UX research, and formal trade-off escalation part of experiment design. Expedia’s program, which ran more than 1,500 A/B tests annually, also relied on an experimentation playbook that required predefined primary, secondary, and guardrail metrics to reduce cherry-picking and p-hacking. At Twilio, experimentation must account for smaller samples, account-level versus user-level randomization, and interference risks when multiple developers from the same company may see inconsistent experiences or prices. Lau argues that when statistical significance is difficult to reach, teams can still make disciplined decisions based on clearly defined hypotheses, signals, guardrails, and explicit conviction rather than treating significance as the only basis for action. She identifies self-service experimentation through Amplitude and the integration of web, financial, and account data as key opportunities to improve velocity and better understand customer journeys and friction points.
No tracked trend matches for this post yet.
Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.