I want it that way: Building experimentation infrastructure and culture with Ronny Kohavi

A recent virtual meetup featured insights from Allon Korem, CEO of Bell, and Ronny Kohavi, a renowned expert in experimentation, focusing on key areas essential to effective A/B testing: infrastructure, experimentation culture, and learning from failures. They emphasized the importance of building the right foundation, fostering a culture that embraces risk-taking and learning from outcomes, and understanding that failure is an integral part of the process. Ronny shared his experience at Microsoft, highlighting the significance of having effective PMs and acknowledging that only 33% of ideas were successful, stressing the need to accept failure as a norm. Organizations were advised to recognize their maturity level in experimentation, balance short-term and long-term goals, and not aim for the highest level of maturity immediately. Misinterpretations of p-values and the normalcy of high failure rates were discussed, along with strategies such as building features on a small scale initially. The conversation also touched on the widespread use of A/B testing in Israel, the importance of addressing sample ratio mismatches, and the benefits of using experimentation tools. During the Q&A, topics such as Bayesian vs. frequentist approaches, multivariable vs. multivariate testing, A/A tests, and asymmetric allocation were discussed, providing a comprehensive overview of best practices for successful experimentation.