September 2024 Summaries

Statsig experienced inefficiencies with their initial Pod Disruption Budget (PDB) setup in Kubernetes, especially for services with fewer than five pods, where the minAvailable setting effectively locked all pods from disruptions, leading to resource waste and blocked updates. This setup was problematic during low traffic periods and time-sensitive updates, such as security patches, because it caused outdated configurations to persist. To address these issues, Statsig switched from using minAvailable to maxUnavailable in their PDB configuration, allowing at least one pod to be disrupted even in smaller deployments. This change improved resource management and deployment velocity by preventing resource waste and facilitating smoother rolling updates. The updated configuration, which specifies maxUnavailable, enables controlled and flexible handling of disruptions and is recommended for teams managing Kubernetes service deployments with variable pod counts.

The text presents a satirical perspective on the drawbacks of using experimentation tools in product development, emphasizing the appeal of rapid deployment over meticulous planning and data analysis. It humorously critiques the time-consuming nature of setting up experiments, the difficulty in selecting appropriate metrics, the complexity and cost of integrating such tools into the tech stack, and the uncomfortable truths that data might reveal about beloved features. Instead, it suggests embracing uncertainty and relying on instinct, highlighting the thrill of quick decision-making and the excitement of dealing with issues as they arise. The narrative concludes with a tongue-in-cheek encouragement to adopt a more reckless approach to shipping products, while also offering a subtle nod to the potential benefits of using a tool like Statsig for those who do see value in data-driven insights.

Turning an idea into a real product requires significant craftsmanship and technical expertise, which can be enhanced by data-driven product development tools like Statsig. Statsig simplifies the process by allowing users to experiment and make informed decisions without extensive coding or infrastructure management. This platform democratizes advanced product development tools, traditionally used by big tech companies, making them accessible to all. Statsig's features, such as feature gates, dynamic configurations, and A/B testing, enable users to test new ideas safely, measure impact with built-in analytics, and confidently deploy changes. The platform's flexibility allows users to either utilize Statsig Cloud or integrate it within their existing data infrastructure, providing scalability and reliability for any workload size. Through practical examples, like e-commerce personalization and OpenAI's API feature testing, Statsig demonstrates its ability to enhance the product development process by reducing risk and improving decision-making efficiency.

The statement "You can never accept the null hypothesis! You can only fail to reject it!" is a misunderstanding rooted in the conflation of Fisher's significance testing with Neyman-Pearson's hypothesis testing. Fisher's framework focuses on using p-values to measure evidence against a null hypothesis without specifying an alternative, while Neyman-Pearson's framework treats both null and alternative hypotheses symmetrically and uses error rates to make clear decisions between them. The misconception arises because Fisher's approach does not support "accepting" the null hypothesis due to the lack of a definitive proof, whereas in Neyman-Pearson's framework, "accepting" a hypothesis is necessary as part of a decision-making process that temporarily assumes one hypothesis as true. The key difference lies in Fisher's emphasis on assessing evidence without fixed significance levels, versus Neyman-Pearson's structured decision-making that controls error probabilities and uses critical regions instead of p-values. Understanding these frameworks as distinct, akin to different sports, clarifies why both "accepting" and "rejecting" hypotheses have their place within statistical analysis depending on the framework employed.

Statsig has implemented an optimization to reduce config propagation latency, focusing on the time it takes for configuration changes to reflect in frontend or backend systems through their SDKs. This improvement centers around the use of "target apps," an advanced feature that manages which configurations are sent to specific SDK instances, enhancing performance and security by filtering out irrelevant configurations. By pre-filtering configurations based on their relevance to target apps, Statsig has improved Node.js event loop times, achieving a 66% reduction in config payload generation time for one of their major customers, thereby lowering latency and enhancing system responsiveness. The optimization involves an in-memory cache that maps configurations to target apps, ensuring only necessary configurations are included, which is crucial for companies with multiple apps and services. This advancement supports performance improvements by reducing memory usage and initialization times while enhancing security by preventing exposure of sensitive configurations.

Feature flag tools are essential for managing feature rollouts and experimentation, but selecting the right option can be challenging due to varying pricing models. A spreadsheet has been created to compare the costs of different platforms, with Statsig standing out as the only provider offering free feature flags for self-service companies at all usage levels. In contrast, other platforms like LaunchDarkly and PostHog have free tiers that become costly as user numbers increase. The analysis uses Monthly Active Users (MAU) as a benchmark to compare pricing, based on assumptions such as the number of sessions and gates per user. While cost is a significant factor, other important considerations include scalability, security, and features like flag lifecycle management and impact analysis. The document encourages reviewing the detailed methodology for further understanding and offers assistance for inquiries about feature flagging platforms.

Statsig's quarterly hackathons are lively events where employees have the freedom to explore creative ideas, leading to the development of innovative projects and features, some of which may eventually become part of the company's offerings. During the Q3 2024 hackathon, several AI-driven projects were introduced, such as an Experiment Ideas Generator, an AI Console Search, and Codemonkey, a coding assistant leveraging large language models. Other noteworthy projects included CUPAC, a predictive tool for user behavior, and Sigma, a new read/write cache system designed to enhance caching efficiency. Additionally, the hackathon featured creative cultural initiatives, like the Statsig Dog Calendar and Statsig Wrapped, reminiscent of Spotify Wrapped but for experimentation. While many projects remain conceptual, the hackathon showcases the enthusiastic and innovative spirit at Statsig, highlighting both potential new features and the vibrant company culture.

Session replay tools enable the capture of user interactions on websites, and a recent analysis highlights Statsig as the most cost-effective option across various usage levels up to 100,000 sessions per month. The study involved a detailed cost comparison among several vendors, revealing that LogRocket and Hotjar are more expensive for sites with over 5,000 sessions, while Statsig offers a free tier with significantly more sessions than PostHog. The comparison used standardized assumptions, focusing on daily and monthly session pricing models, with daily models used by platforms like Hotjar and monthly models by Statsig and others. Beyond cost, it is crucial to consider additional features such as heat maps, user feedback tools, and integration capabilities, as well as the platform's scalability and performance to ensure it meets traffic demands effectively. The analysis encourages further review of the detailed methodology provided and suggests that Statsig’s free tier can be a beneficial starting point for those seeking comprehensive insights into user behavior.

Funnel metrics are essential tools in analytics platforms for tracking user behavior and identifying where users drop off during their interaction with a product, yet they have been underutilized in experimentation platforms due to complexity and perceived limitations. Despite initial skepticism, Statsig has enhanced its funnel functionality to provide clearer insights into user actions by measuring the sequence of events per user or session, which can help diagnose issues in conversion rates. While funnels can be complex and are often treated with insufficient statistical rigor, their proper use as diagnostic tools can provide a deeper understanding of experiment results, especially in growth scenarios. Effective funnel usage involves setting clear parameters, understanding user versus session-level data, and ensuring statistical robustness in conversion metrics. Statsig emphasizes the importance of ordered events, multi-step funnels, and session breakdowns for a comprehensive analysis, and encourages the adoption of best practices to maximize the potential of funnels in experimentation.

CUPED is an algorithmic tool used to increase the speed and accuracy of experimentation programs by leveraging pre-experiment data to explain away variance in result data. It helps reduce standard error, making it possible to observe statistically significant results even with relatively small effects. CUPED works by normalizing post-experimental values based on pre-experimental metrics, allowing for a more accurate estimation of the experiment effect. The algorithm is most effective when applied to existing user experiments with access to historical data, and its ability to adjust values is based on the correlation between a metric and its past value for the same user. CUPED has been successfully used in various companies, including Facebook and Optimizely, to improve experimentation outcomes.

A recent virtual meetup featured insights from Allon Korem, CEO of Bell, and Ronny Kohavi, a renowned expert in experimentation, focusing on key areas essential to effective A/B testing: infrastructure, experimentation culture, and learning from failures. They emphasized the importance of building the right foundation, fostering a culture that embraces risk-taking and learning from outcomes, and understanding that failure is an integral part of the process. Ronny shared his experience at Microsoft, highlighting the significance of having effective PMs and acknowledging that only 33% of ideas were successful, stressing the need to accept failure as a norm. Organizations were advised to recognize their maturity level in experimentation, balance short-term and long-term goals, and not aim for the highest level of maturity immediately. Misinterpretations of p-values and the normalcy of high failure rates were discussed, along with strategies such as building features on a small scale initially. The conversation also touched on the widespread use of A/B testing in Israel, the importance of addressing sample ratio mismatches, and the benefits of using experimentation tools. During the Q&A, topics such as Bayesian vs. frequentist approaches, multivariable vs. multivariate testing, A/A tests, and asymmetric allocation were discussed, providing a comprehensive overview of best practices for successful experimentation.

Pricing experimentation platforms can be complex due to varying pricing structures and usage tiers, prompting the creation of a detailed pricing model to clarify costs across popular platforms. The model uses assumptions such as Monthly Active Users (MAU) as a standardized benchmark and factors like session and event metrics to enable comparison. Statsig is identified as the most cost-effective and scalable option, especially at higher volumes, with enterprise pricing starting at around 200K MAU, while Posthog is noted as the most expensive with a restrictive free tier. The analysis emphasizes the importance of scalability and bundled discounts when evaluating platforms, as some, like Statsig, offer comprehensive features including advanced experimentation and unlimited feature flags at no extra cost. Understanding these cost dynamics aids in making informed decisions regarding platform selection, and the authors encourage personalized discussions to tailor the best choice for individual needs.

The narrative describes the author's journey of understanding the significance of A/B testing through experiences at Facebook, particularly with the Marketplace initiative. Initially enthusiastic about incorporating e-commerce into Marketplace, the author recounts a significant failure revealed by an A/B test, which showed a decline in key metrics following a major launch. This led to questioning whether the failure was due to the idea itself or its execution. A subsequent experiment using image background variations provided critical insights, revealing that Marketplace users were more interested in finding deals than engaging with traditional e-commerce products. This experiment demonstrated the power of A/B testing in guiding strategic decisions and settling debates, ultimately refocusing efforts on local listings with successful results. Through this process, the author gained a deeper appreciation for data-driven decision-making, highlighting how A/B testing can effectively challenge assumptions and redirect strategies.

After returning from South Korea, the author began their journey at Statsig, quickly diving into onboarding tasks and meeting colleagues in an environment that fostered teamwork and creativity. Over the first few months, they worked on impactful projects, including revamping the onboarding process, which resulted in a 10% increase in successful completions, and developing a Beta Testing feature that leveraged feature flags for user enrollment. With a supportive engineering culture, they appreciated the rapid iteration and customer-focused development that allowed them to implement changes swiftly, enhancing user experience. Participating in company events like a hackathon, laser tag, and a summer picnic provided opportunities for bonding and learning, while tackling challenges such as improving the analytics dashboard demonstrated the company's commitment to innovation. These experiences highlighted the author's growth and the value of being part of a dynamic team, reinforcing their decision to join Statsig and their anticipation for future contributions.

A/B testing, grounded in the principles of randomized controlled trials (RCTs), is a crucial method for establishing causality and measuring effectiveness in various fields, including product development. By utilizing randomization, A/B testing mitigates selection bias and ensures that differences between treatment and control groups are due to the treatment itself, rather than external factors. This method allows companies to accurately assess the impact of changes, such as new features, on key metrics, as demonstrated by Recroom's experience of identifying a significant metric drop after a UI revamp. A/B testing reveals that a majority of ideas may not produce the desired outcomes, emphasizing the importance of measurement and iteration in development processes. Overall, A/B testing helps overcome human biases and errors in attribution, providing a reliable framework for making informed decisions based on evidence.

Boeing's recent safety issues, including an emergency involving an Alaska Airlines flight and whistleblowers highlighting safety concerns, have led to increased consumer wariness and queries about aircraft models. In response, travel companies and platforms like Kayak have adapted to these concerns by promoting existing features that allow users to filter out Boeing planes from their travel bookings. This strategic move reflects Kayak's agility in addressing user sentiment by repositioning its aircraft filter feature to meet the growing demand for safer travel options. The decision to elevate this feature was driven by user engagement data, demonstrating Kayak's effective use of existing resources and data-driven strategy to enhance the user experience amid rising safety concerns.

Statsig is revamping its design system with the introduction of the Pluto design system to create a more focused and approachable platform for its growing user base. This redesign aims to simplify user experience by making the interface more intuitive, seamless, trusted, delightful, and scalable. Key updates include improved navigation, new layouts for better data visualization, a streamlined creation modal, and an enhanced dark mode. The design system's consistent and scalable components are intended to support future growth and maintain usability as new features are added. The Pluto design system's rollout began in August, with plans for a phased introduction accompanied by detailed guides and support to ensure a smooth transition for all users.