March 2022 Summaries

Statsig and mParticle can now be seamlessly integrated, enabling automatic ingestion of mParticle events into Statsig for enhanced feature management and product experimentation. This integration facilitates running a significantly higher number of experiments by connecting mParticle events to Statsig experiments without additional effort. Users gain the ability to compare how test and control groups affect mParticle events, providing a comprehensive understanding of feature impacts. The integration allows real-time event analysis, assisting in identifying features that drive the most significant changes, either positively or negatively, in mParticle metrics. Configuration is straightforward, utilizing Statsig's documentation and mParticle's integration directory. Additionally, insights into experimentation culture, CUPED methodology, and learnings from industry leaders offer valuable context for users looking to deepen their understanding of A/B testing and experimentation strategies.

Statsig has introduced updates to its Stats Engine used in Pulse, aimed at improving the accuracy and clarity of experimental analysis. The major changes include shifting the unit of analysis from a user-day to a user, and adopting Welch's t-test for small sample sizes, which enhances the confidence intervals and reduces biases in both long-term and small-scale experiments. This update has been applied to all ongoing and future experiments, with past experiments retaining the old engine's results. While most results remain unchanged, some may experience a shift in statistical significance. Additionally, the new engine reduces false positive rates but limits flexibility in date range selection, offering predefined time windows and customizable queries for data analysis. Statsig continues to seek user feedback and engagement to further refine their tools and supports inquiries through various communication channels.

Switching to a newer version of an internal SDK led to unexpected increases in gaming sessions on Facebook, driven by crashes that prompted users to restart games, demonstrating the complexities and surprises inherent in experimentation. This highlights the importance of a robust experimentation culture to counteract biases and ensure valid results, emphasizing the need for a scientific approach to testing and review processes. Companies like Facebook, Spotify, and Amazon have developed sophisticated practices to run numerous parallel experiments, prioritizing speed of learning and comprehensive reviews by diverse teams to avoid confirmation bias and p-hacking. Establishing an effective experimentation culture involves active reinforcement, hypothesis-driven testing, and insightful reviews, both online and offline, to build a learning organization that can adapt and evolve from both successful and unsuccessful experiments. These practices contribute to long-term insights that shape future priorities and guardrail metrics, with examples such as Amazon's streamlined checkout processes and Facebook's friend connection strategies illustrating the value of lessons learned.

Two sales professionals from Statsig, both former Snowflake employees, have developed an essential business technology stack comprising key tools like Sales Navigator, ZoomInfo, Outreach, and Calendly to support their sales efforts. Sales Navigator is pivotal for outreach and targeting the ex-Facebook community, while ZoomInfo provides invaluable contact data, especially for tracking personal emails. Outreach facilitates large-scale outbound activities with features for automating sequences and A/B testing, though it is less practical for account executives who focus less on cold outreach. Additionally, Calendly simplifies scheduling across different time zones, proving particularly useful for international collaboration. Seamless.ai complements ZoomInfo by predicting emails for small-to-medium businesses, offering robust coverage when used together. The narrative also touches on the importance of experimentation in Statsig's culture, with references to A/B testing and insights from industry experts like Ronny Kohavi and Allon Korem.

The text highlights misconceptions in product experimentation, emphasizing that traditional academic methods can be outdated and inefficient. Companies like Spotify, Airbnb, Amazon, and Meta have developed practical approaches that allow them to conduct experiments more rapidly, prioritizing speedy learning over precision. Statsig aims to bridge the gap by offering tools that enable companies to quickly analyze experiment results. However, there's a caution against misinterpreting data due to factors like novelty effects, seasonality, and biases in short-term data. The importance of a structured approach—having clear hypotheses, corroborating results, and understanding unexpected outcomes—is stressed to avoid erroneous conclusions. Additionally, the text underscores the need for a solid experimentation culture and infrastructure, citing insights from industry experts. It briefly touches on the evolution of platforms like Optimizely and Facebook's strategic transformations through A/B testing, and concludes with a nod to Statsig's innovative culture and the value of A/B testing in evidence gathering for both new and experienced experimenters.