June 2023 Summaries

Random selection in experiments can lead to false positives, as a 95%-confidence frequentist analysis might produce them in 5% of comparisons, potentially resulting in random groups that differ by chance before any intervention. Statsig addresses this issue by proactively detecting and flagging pre-experiment bias, ensuring trustworthy results. While tools like CUPED can adjust data for pre-experiment bias, they have limitations, such as not fully accounting for bias or being inapplicable to certain metrics. Statsig's approach involves scanning for pre-experiment bias in Scorecard Metrics using a sensitive p-value and notifying experiment owners when significant differences are detected. This allows for timely corrections, such as re-salting suspect experiments, and helps balance the need for alerting and identifying genuine issues. The integration of bias detection in Statsig experiments promises users confidence that their experiments are not affected by pre-existing random bias, enhancing the reliability of A/B testing outcomes.

Online experimentation, particularly A/B testing, is often celebrated for its potential to produce significant improvements in product performance, yet in practice, the true value lies in preventing losses rather than achieving major wins. While legendary success stories of minor changes leading to substantial boosts in metrics capture attention, the reality is that most experiments fail to achieve their goals, and the majority of observed "wins" may be misleading due to factors like Twyman's Law, novelty effects, or poor evaluation criteria. It is crucial to maintain skepticism towards positive results and prioritize understanding the underlying causes of metric changes. Preventing negative outcomes is as important as pursuing positive ones, as demonstrated by Rec Room's experience of turning a potential 35% regression into a 10% improvement. The strategic focus should be on using experimentation to holistically assess all changes, identify and mitigate regressions using strong guardrail and ecosystem metrics, and celebrate the prevention of losses alongside the pursuit of wins. This approach not only requires robust experimental infrastructure but also a recalibration of how success is measured and celebrated in the context of product development.

Dynamic Configs offer a streamlined approach to managing inputs for large language models (LLMs) by centralizing control over various parameters, which simplifies the process of experimenting with different models and making upgrades. By using Statsig's interface, users can map LLM inputs and outputs to dynamic configs, allowing for easy integration and adaptation of data from various sources, like user behavior or system logs, into a cohesive system. This method not only cleans up code but also facilitates targeted user experiences, such as localizing responses for different language groups. Additionally, Statsig provides resources for experimentation, offering insights into best practices for A/B testing and fostering a strong culture of innovation and learning from failures.

In recent years, the market for feature flags, experimentation, and analysis has expanded, with several platforms like Eppo, Statsig, Amplitude, Split, and Posthog emerging as key players. Eppo, founded in 2021, offers a warehouse-native experimentation platform that integrates with data warehouses like Snowflake and BigQuery, but lacks robust collaboration features and real-time updates. Statsig, founded by ex-Meta engineers, provides both cloud-based and warehouse-native solutions with a straightforward pricing model, though its product diversity can be overwhelming. Amplitude focuses on digital analytics with real-time monitoring but is costly and complex for smaller teams. Split and Posthog offer user-friendly and developer-focused solutions, with Posthog being an open-source platform. Despite their individual strengths and weaknesses, these platforms cater to different business needs, from startups to large enterprises, emphasizing the importance of selecting the right tool based on specific requirements and scale.

Observability, originally rooted in Control Theory, has evolved significantly since the 1980s as software and computer systems became more complex, requiring advanced monitoring of key outputs to track performance and catch errors. The rise of CI/CD, microservices, and distributed architectures in the 2000s increased the demand for real-time monitoring and led to the development of sophisticated logging, metrics, and tracing tools. Companies like Datadog, Honeycomb, and New Relic emerged to address these challenges, focusing on understanding system states and troubleshooting. However, traditional observability is largely reactive and infrastructure-focused, prompting a shift towards "product observability," which emphasizes understanding user interactions and correlating them with system behavior. This approach, advocated by Statsig, involves a proactive, adaptive, and continuous measurement of product impact using a comprehensive set of tools, enabling teams to track metrics, conduct experiments, and communicate insights effectively. Product observability aims to enhance user-centered design, reduce risks in product changes, and foster collaboration across teams, ultimately leading to improved product quality and user satisfaction. The future of software development is likely to see product observability becoming ubiquitous, as it offers significant benefits in terms of agility, efficiency, and overall success.

Statsig has launched Statsig Warehouse Native, a tool that integrates with data warehouses like BigQuery, Snowflake, Databricks, and Redshift to enhance experiment analysis directly within these systems. This new product allows companies to run sophisticated experiment analyses using existing datasets and Statsig's robust statistics engine, which features CUPED and sequential analysis capabilities. It offers no-code tools for segmenting and detailed results analysis, supporting feature flags, remote configuration, and software release management through integrated SDKs. Warehouse Native aims to streamline experimentation workflows by providing full visibility into feature impacts and eliminating the need for redundant metric computations. By connecting to data warehouses, it allows product and data teams to make data-driven decisions more efficiently, with the flexibility to integrate with various existing tools for event logging and metric computation. Statsig also emphasizes transparency by providing access to SQL queries and statistical methodologies, aiming to enhance experimentation culture and infrastructure for developers, data scientists, and product managers, inspired by insights from industry leaders and experiences from companies like Facebook.

The conversation between Tim from Statsig and Ronny Kohavi explores the evolution and current landscape of experimentation in product development, highlighting the increased presence of experimentation platforms compared to the mid-2000s when Kohavi worked at Amazon. Kohavi emphasizes the importance of trust, rigorous testing methods like A/A tests, and the challenges organizations face due to inertia in implementing these practices. He notes common mistakes in experimentation, such as premature celebration of results and lack of adequate training, and discusses the need for a cultural shift toward data-driven decision-making. Kohavi also addresses the role of AI in experimentation, suggesting that it will enhance segment identification and parameter optimization, while emphasizing the integration of feature flags and controlled experiments for more sensitive evaluation. He underscores the significance of server-side experimentation over client-side due to scalability concerns. Kohavi reflects on the cultural challenges in adopting experimentation, drawing parallels to paradigm shifts described by Thomas Kuhn, and concludes by advocating for continuous learning and innovation in the field.

Building a successful AI product in a non-AI company involves identifying a compelling problem that generative AI can solve for users, creating a viable first version using proprietary models enhanced with private data, and continuously improving the product through user feedback and experimentation. Unlike the previous generation of AI/ML features, which relied on unique models for differentiation, the current wave of AI emphasizes the application of context and data unique to a company, allowing even non-AI companies to develop impactful AI features. Notion serves as a prime example of this approach, having leveraged customer pain points to integrate AI into its existing application, enhancing functionality with contextually aware features that continue to evolve. Engaging users early through alpha and beta testing phases, and measuring key metrics such as engagement and latency, are crucial steps in refining the product. A culture of experimentation and iterative improvement, supported by a robust measurement system, enables companies to adapt swiftly to advancements in AI technologies and maintain competitive advantage.

Statsig provides a comprehensive suite of user accounting metrics designed to assist growth teams in tracking and optimizing user behavior effectively. By integrating Statsig's SDK into a product, companies can automatically compute key metrics like daily, weekly, and monthly active users, new users, and user stickiness, which are crucial for understanding business direction, user engagement, seasonality, and retention. These metrics not only help in identifying effective re-engagement tactics and onboarding strategies but also provide insights into long-term retention, which is essential for achieving product-market fit. Statsig simplifies the process of defining an "active user" and allows for the easy calculation of these metrics, which were traditionally challenging and required advanced data science expertise. This approach empowers teams to track feature impacts and experiments with granular visibility, ensuring that no team is left without critical growth data when launching new products or businesses.

The Seattle AI event showcased a diverse panel of experts discussing the evolution and future of AI, highlighting how recent advancements in AI models and tools have democratized the development of AI-powered applications. The panelists, including leaders like S. Somasegar from Madrona and Alessya Visnjic from WhyLabs, explored how the accessibility of foundation models has drastically increased the capacity for companies to build AI applications, shifting the paradigm for developers. They noted that while large companies like OpenAI might dominate the model API market due to their resources, open-source models and specialized tools could offer competitive alternatives for smaller entities. The discussion underscored the importance of tools in deploying and managing AI models, regardless of the approach companies take, and emphasized the ongoing excitement and innovation within Seattle's AI community. The session also touched upon the significance of robust experimentation and testing in AI development, as illustrated by experiences shared from Statsig's projects and culture.

Experimentation in product development involves a shift from focusing on academic statistical debates to addressing practical concerns like best practices and feedback mechanisms, as observed by many companies including Meta and Amazon. Effective experimentation is likened to manager training, where peer reviews and critiques are essential for calibrating expectations and learning. This approach mirrors the scientific method, integrating skepticism and peer review to mitigate biases such as confirmation bias and p-hacking. Experimenters are encouraged to document and review their plans and outcomes, fostering organizational learning and distributing best practices. Insights gained from experiments, whether successful or not, help shape future priorities and metrics, as exemplified by Amazon's and Facebook's well-documented findings. The ultimate goal of experimentation is to cultivate a learning organization, with companies like Statsig offering a culture that supports hackathons and innovative projects to reinforce this ethos.