May 2022 Summaries

The text discusses the development and functionality of a macro tool initially used within Facebook's Workplace, which allowed employees to efficiently use images or gifs across various internal platforms by typing a simple command. This tool was valued for its speed, ease of use, and entertainment factor, leading to the recreation of a similar tool named "marcos" at Statsig. The process of setting up and integrating this tool involves specific technical steps, including creating custom integrations and utilizing webhooks, with a notable mention of using ngrok for testing. The text also highlights challenges faced due to limitations in Workplace’s custom integration capabilities and suggests potential improvements for broader application. The narrative further touches on the broader context of innovation and experimentation within the company, noting the impact of A/B testing and the evolving landscape of web experience platforms.

In just over a year since joining the early-stage startup Statsig, the author reflects on their journey from leaving Facebook to becoming a crucial part of Statsig's growth and success. Initially joining as the 9th employee and first design hire, they contributed significantly to the company's rapid development, which saw Statsig grow from stealth mode with no customers to serving over 300 companies, including well-known brands like Headspace and Notion. The company has achieved remarkable milestones, such as closing a $43M Series B funding round led by Sequoia Capital, boosting its valuation to $420M, and expanding its team to over 30 members. The author shares personal experiences, from building the Statsig Design System to engaging in team activities and celebrating achievements, while emphasizing the strong team culture and learning opportunities at the startup. They express excitement for the future, acknowledging the challenges and opportunities that come with startup life and the ambition to shape the experimentation culture for many companies.

The text explores the importance of time series analysis in online experimentation, emphasizing its role in revealing insights that might be obscured in aggregated data, such as novelty effects and pre-experiment bias. It highlights how time series charts, particularly those showing metric impact by user tenure, can validate or challenge assumptions about experimental results. The text stresses the significance of understanding daily variations to identify anomalies or false positives, and it explains how cumulative time series can help determine the stability of results and the potential need for extended experimentation. Additionally, it cautions against over-reliance on slicing data, which can lead to p-hacking, while advocating for a balanced approach to data interpretation to inform decision-making. This comprehensive guide underscores the value of time series analysis in refining experiment strategy and ensuring robust, evidence-based conclusions.

The text humorously explores decision-making in road rage incidents using Bayesian statistics to determine whether a driver who behaves poorly is likely a bad driver or simply made a mistake. It equates this decision-making process to statistical experimentation, highlighting concepts such as Type 1 and Type 2 errors, and the use of Bayes’ theorem to update prior beliefs based on new evidence. The discussion extends into the realm of A/B testing, emphasizing the importance of understanding success rates, significance levels, and power levels to accurately interpret experimental results and avoid false positives or negatives. The narrative also touches on experimentation culture and infrastructure, with a nod to companies like Statsig striving to balance ease of use in experimentation tools with the ability to deeply understand the results.

The text discusses the importance of experimentation and holdouts in product development, particularly within fast-moving teams like Facebook, to ensure long-term success and mitigate over-optimization for short-term gains. It highlights how holdouts, where a small percentage of users are excluded from certain features, allow teams to measure the true impact of those features by comparing metrics over time. By creating holdouts, teams can separate the effects of external factors from feature launches and evaluate both the cumulative and long-term impacts of their developments. However, the text also notes that holdouts can be costly and complex, requiring careful consideration of their size, duration, and purpose to avoid unnecessary expenses or skewed results. Additionally, the text touches on the advantages of simpler testing methods like backtests and emphasizes a thoughtful approach to experimentation, urging teams to start with straightforward holdouts before moving to more sophisticated methods.