March 2023 Summaries

Statsig has launched a new virtual event series called Learning Labs, which offers users the chance to explore niche topics and integrate specific workflows into their testing programs, including the application of machine learning to feature flag infrastructure. Led by Statsig software engineer Tyler VanHaren, participants can learn how to enhance feature flags using machine learning to improve testing programs and enrich ML models. The series also covers topics like CUPED, a method for running experiments more efficiently and with less bias, and includes insights from experts Ronny Kohavi and Allon Korem on fostering a robust experimentation culture. Additionally, the text reflects on the evolution of A/B testing and experimentation platforms like Optimizely and discusses how causal evidence and A/B testing have influenced product strategies at Facebook. The author shares personal experiences from the initial months at Statsig, highlighting the company's unique culture marked by innovation and collaboration.

Feature gates, also known as feature flags, are influential tools in software development, allowing developers to control the deployment of features and manage different code paths within an application. While they offer significant flexibility, they also introduce complexity, as maintaining multiple paths can complicate the codebase. To address this, developers should regularly review and remove obsolete feature gates after features are fully deployed or discontinued. A new system helps differentiate between permanent and temporary gates, marking stale ones for removal when certain conditions are met, like having been fully rolled out or lacking recent activity. Teams can incorporate regular reviews of feature gates into their workflows, such as weekly check-ins or quarterly cleanups, to ensure code clarity and efficiency. Additionally, insights from industry experts and examples from companies like Facebook highlight the importance of rigorous experimentation and learning from testing failures to foster a strong culture of innovation and adaptability.

Generative AI's evolving capabilities, exemplified by large foundation models like OpenAI's ChatGPT, have significantly increased the scope for developing innovative applications. Statsig, an experimentation platform, has seen a surge in interest from companies looking to refine AI-based features by testing different parameters and configurations. Notable examples include Captions, WhatNot, and Notion, which have leveraged Statsig to enhance their AI offerings. The platform simplifies experimentation with generative AI, allowing developers to test and compare the efficacy of various foundation models and their configurations in a single environment. This capability is crucial as it provides visibility into the effects on user engagement and model performance, facilitating informed decision-making without unintended consequences. Statsig's ability to conduct simultaneous experiments on app features and foundational model changes makes it a powerful tool for developers, enabling them to innovate rapidly while maintaining control over the impact of their adjustments. As more companies delve into AI applications, Statsig's platform offers a structured approach to experimentation, ensuring that new features are backed by empirical data and insights.

After spending nearly eight years in big tech companies like Facebook and Uber, the author decided to transition to an early-stage startup environment, driven by a desire to tackle more diverse problems with fewer resources. This shift, which came during the global pandemic, was guided by the author's "people-to-problems ratio" heuristic, pointing them back to their original plan of working in startups. At a startup, the author discovered the importance of team dynamics, the flexibility required in product management, and the necessity of taking innovative "shots on goal" to address significant challenges. This new chapter involved overcoming perfectionism, embracing rapid iterations, and understanding the different roles founders can play in product strategy. Through experiences at Statsig, the author found fulfillment in the close-knit team environment, despite initial uncertainties. They emphasize that transitioning from big tech to a startup can enhance a product manager's skills and career, offering valuable insights into startup culture, experimentation, and product development.

DispatchHealth, a platform providing on-demand medical care at home, faced challenges in prioritizing high-impact product features due to outdated product development processes and technical debt. The introduction of experimentation, particularly through an A/B test of a 'Click to Call' feature, marked a fundamental shift in their approach. Although the feature aimed to improve patient conversion rates by streamlining the user experience, it unexpectedly led to increased missed patient rates, revealing critical operational and technical gaps. This outcome demonstrated the value of experimentation in identifying issues and informed future strategic decisions. Despite the initial disappointment, the experience highlighted the significance of building a robust experimentation culture and infrastructure, with support from partners like Statsig, to enhance product strategy and analytics capabilities.