The pros and cons of A/B testing

Linear's CEO, Karri Saarinen, sparked a debate by stating that his team avoids A/B testing, favoring intuition and feedback over data-driven decisions for long-term strategy. Saarinen argues that relying on taste and opinions fosters creativity, whereas A/B testing can lead to over-optimization and a focus on efficiency at the expense of addressing customer needs. Despite the widespread use of A/B testing for its quick, definitive, and flexible insights, it presents limitations such as false positives and inefficacy in addressing strategic questions. While A/B testing can validate ideas rapidly and influence stakeholder buy-in, it often falls short in understanding customer needs without qualitative research. The discussion highlights the need for a balanced approach, incorporating A/B testing as part of a diverse research strategy while allowing for creative autonomy and direct customer engagement to truly address user needs.