August 2023 Summaries

The text discusses the concept of Sample Ratio Mismatch (SRM) in experimental design, particularly in the context of random assignment issues, using coin flip experiments as a metaphor to explain the statistical anomaly. It describes how SRM can occur due to various factors such as non-random assignment, data processing issues, or differences in exposure between test and control groups. The text highlights the importance of checking for SRM using tools like chi-squared tests and emphasizes the role of the p-value in identifying significant deviations from expected results. It further explores how tools like Statsig can automate the detection of SRM and suggests strategies to address and debug SRM, such as re-running experiments or employing methods like CUPED to control for pre-experiment bias. The text underscores the necessity of understanding the root cause of SRM to make informed decisions about experiment results and emphasizes the importance of rigorous methodologies to ensure reliable experimental outcomes.

Amazon is renowned for its strategic use of experimentation, not only in optimizing its user interface through button placement and design but also in exploring deeper business strategies that extend beyond UI adjustments. The company's experimentation ethos is evident in initiatives like Amazon Prime, where the balance between customer satisfaction and conversion rates is analyzed, as well as in dynamic pricing algorithms that adjust in real-time based on demand and market conditions. This approach also encompasses recommendation systems, payment, and shipping options, all aimed at enhancing the customer experience while maximizing revenue. These experiments highlight the importance of testing broader business hypotheses, suggesting that nearly every major decision can be refined through experimentation. The text emphasizes that while UI changes can be impactful, the true potential of experimentation lies in its ability to optimize fundamental business operations, as illustrated by Amazon's various strategies and insights from industry experts.

In late August, a bug affecting the resolution of Feature Flags using the JS SDK in recent Chrome versions emerged, causing unexpected default values to appear without any code changes on the affected platform. The issue was traced back to a map lookup problem in the js-sha256 library, which had not been updated in years. This bug, reported by users and confirmed through reproducible code on GitHub, was due to a recent Chrome JIT compiler optimization error that incorrectly handled a specific variable. The mitigation involved using the problematic variable in a closure to prevent its premature optimization, and a patch for the JS SDK was released on August 27 to address the issue. Additionally, the text discusses the use of CUPED for faster, less biased experiments and highlights insights from Ronny Kohavi and Allon Korem on experimentation culture and infrastructure. It also touches on the evolution of the web experience platform Optimizely, shifts in Facebook's product strategies due to A/B testing, and the unique culture at Statsig, emphasizing the reliability of A/B testing for gathering evidence.

The recent in-person Experimentation Meetup at the Statsig office in Bellevue, WA, featured Ronny, who shared insightful experiences from his tenure at Microsoft, Amazon, and Airbnb. This collaboration was so successful that it led to a virtual AMA with Ronny and Tim Chan. The event highlighted the significance of CUPED in running experiments more efficiently, along with insights from Ronny Kohavi and Allon Korem on fostering a robust experimentation culture and infrastructure. It also touched on Optimizely's evolution as a major web experience platform and the transformative impact of A/B testing, exemplified by Deltoid's influence on Facebook's product strategies. The text concludes with an emphasis on the reliability of A/B testing for gathering evidence, appealing to both seasoned and novice experimenters.

At the inaugural Product Growth Forum in San Francisco, hosted by Statsig in collaboration with Ben Gilbert and David Rosenthal of the Acquired Podcast, industry leaders including Vijaye Raji of Statsig, Karandeep Anand of Brex, Laura Jones of Instacart, and Kris Rasmussen of Figma discussed strategies for rapid and sustainable product growth. The event featured insights into CUPED, a technique to expedite and reduce bias in experiments, as well as lessons from Ronny Kohavi and Allon Korem on fostering a robust experimentation culture and infrastructure. Attendees explored the evolution of A/B testing from Optimizely's pioneering efforts to Facebook's strategic shifts influenced by Deltoid and causal evidence, while personal reflections on the dynamic culture and innovative projects at Statsig highlighted the importance of a strong internal community in driving success.

In B2B SaaS applications, the initial user experience is crucial for engagement and retention, especially within product-led growth (PLG) strategies where the self-service journey's intuitiveness significantly impacts key metrics like daily active users (DAU) and net recurring revenue (NRR). Successful onboarding strategies, exemplified by companies like Canva, HubSpot, and Salesforce, guide users to the "aha moment" by providing personalized, interactive onboarding processes that accelerate time-to-value. These companies' approaches include step-by-step instructions, personalized experiences, and contextual tooltips, which have been developed through experimentation and A/B testing to ensure they enhance user engagement without negative side effects. Statsig's platform supports this by allowing companies to run experiments that measure the impact of onboarding features on core business metrics, potentially leading to substantial revenue growth.

During a summer internship at Statsig, a Product Design Intern worked on various projects, including a New User Experience (NUX) initiative, to understand and improve the product from developers' perspectives, emphasizing the iterative and collaborative nature of design involving product managers, engineers, and other stakeholders. The intern highlighted the importance of design critique sessions, where cross-functional teams align on project directions and navigate the non-linear design process while balancing creativity with consistent frameworks. The internship underscored the value of building personal connections within the company, participating in team-building activities, and maintaining open communication for effective design progress. Additionally, the intern reflected on the significance of creating a strong experimentation culture, learning from failures, and leveraging methodologies like CUPED for efficient A/B testing.

The text explores the decision-making process organizations face when upgrading to a modern experimentation and analytics platform, highlighting the options to build an in-house platform, fork and self-host an open-source solution, or purchase a cloud solution. It emphasizes considerations such as total cost of ownership, opportunity cost, and the ability to trial existing solutions. The challenges of self-hosting, including technical complexities and potential scaling issues, are discussed, suggesting that while open-source solutions may appear cost-effective, they can incur hidden costs and require significant resources. The text advocates for opting for cloud-hosted providers as they are typically more cost-effective and accelerate time to value, especially for companies lacking extensive engineering resources. Additionally, insights into the development of a strong experimentation culture, infrastructure, and the importance of A/B testing are shared, alongside reflections on industry experiences and the evolution of web experience platforms.

Statsig, originally a tool for managing A/B tests and experiments, has evolved into a comprehensive suite of tools utilized by diverse companies for sophisticated experiments, prompting a significant rebranding effort to align its visual identity with its core values of fun, experimentation, and trustworthiness. As Statsig's customer base and audience grew, the company focused on creating a recognizable and consistent brand across various media platforms, utilizing new colors, typography, and iconography to reflect its essence. This rebranding aims to provide a cohesive visual standard for all company assets while continuing to evolve with future growth and changes. The process involved extensive effort from the brand design team, emphasizing continuous learning and adaptation as the company advances. Statsig remains committed to fostering a strong experimentation culture, drawing insights from industry leaders, and maintaining a brand that users can trust across all interactions.

The software development landscape is brimming with companies offering tools for product management, feature flagging, and experimentation, each with unique strengths and weaknesses. Among them, Launchdarkly, founded in 2014, is a SaaS platform known for feature flag management but is gradually branching into experimentation. However, it lacks certain critical features and charges per seat without offering unlimited monthly active users. Competitors such as Statsig provide comprehensive experimentation and feature flagging solutions with transparent pricing and robust features, although the learning curve can be steep for newcomers. Amplitude focuses on analytics with advanced data tracking but is often criticized for its high costs and complexity, making it less suitable for smaller startups. GrowthBook offers an open-source model with transparency in metrics but requires users to ensure scalability. Optimizely, originally a web optimization tool, offers powerful A/B testing capabilities but is often seen as complex and expensive, with hidden costs reported by users. Each of these platforms addresses different needs within the software development ecosystem, offering varied pricing and features to cater to their target audiences.