Starting an experimentation program: Best practices from Yousician

Yousician's journey in building a successful experimentation program highlights several key practices and challenges. Initially tempted to create a custom solution, Yousician opted for Leanplum to leverage its robust AB testing capabilities, gradually increasing the complexity of its experiments. As the program matured, Yousician faced analytics challenges due to the intricacy of full-stack experimentation, which necessitated the development of a custom analytics system for nuanced engagement metrics and lifetime value calculations. The company later transitioned to the Unleash platform to overcome limitations in targeting complexity and consistency, emphasizing the importance of maintaining consistent user experiences across devices. Feature flags played a crucial role not only in experimentation but also in speeding up development processes by allowing integration of incomplete features. Methodology discipline was critical in avoiding pitfalls like p-hacking, while balancing optimization and innovation strategies prevented stagnation. Yousician also addressed the technical debt from accumulated feature flags, emphasizing the necessity of routine cleanup to maintain codebase hygiene. Ultimately, fostering a culture that embraces data-driven decision-making and learning from experiment failures was essential for sustainable experimentation success.