July 2022 Summaries

At the Druid Summit in Seattle 2022, Statsig shared its journey of leveraging Apache Druid to enhance product observability and decision-making through real-time data analysis. The company launched the Events Explorer, a real-time observability tool, utilizing Druid's capabilities for high-volume data ingestion and low-latency queries. Despite the initial complexity of integrating Druid, Statsig benefited from strong community support and detailed documentation, facilitating a smoother onboarding experience. By adopting a metrics-driven approach to optimize Druid configurations, Statsig ensures continuous improvement aligned with product needs. The implementation of Druid, managed via Kubernetes and GitOps practices, allows for flexibility and scalability while maintaining system uptime. Statsig continues to explore Druid's potential for enriched data dimensions and real-time data integration across its products, anticipating further innovations and inviting collaboration from the community.

In the context of product development, determining when to rely on data or research is crucial, and four heuristics can guide this decision: the product stage, user-base size, type of problem (Tylenol vs. Vitamin), and cultural context. Data is beneficial for products with large user bases or post-product/market fit, while research is more effective for smaller user bases or pre-product/market fit to understand user pain points. Tylenol problems, which users are aware of, can be addressed with data and research, whereas Vitamin problems, which users aren't actively seeking solutions for, require a visionary approach. Cultural nuances can render both methods ineffective if not understood, highlighting the need for a combination of data analysis and user research to identify and solve user problems in unfamiliar cultural contexts. These heuristics provide a framework to balance quantitative and qualitative inputs for effective product strategy, adapting to the complexity of real-world scenarios.

In March 2018, amid the Cambridge Analytica scandal, Facebook teams, including the games team, reevaluated their data policies and external developer relationships, leading to significant API and platform changes. During this period, the Instant Games team, where the author worked, conducted audits to decide which APIs to shut down or accelerate the deprecation timeline. The author was responsible for sending emails to developers about these changes, using a feature gate to determine which email each company should receive. However, due to a misconfiguration of the gate's pass percentage, the strict message was mistakenly sent to all companies, highlighting the importance of setting default values appropriately and updating configurations accurately. The narrative also touches on the author's experiences with experimentation culture at Statsig, emphasizing learning from failures and the evolution of product strategies through A/B testing insights.

Statsig has introduced CUPED, a technique to enhance the precision of experimental data by leveraging pre-experimental data to reduce variance and pre-exposure bias, automatically integrating it into customers' experiments. This method, originally publicized by Microsoft, optimizes the evaluation of key metrics by narrowing confidence intervals, lowering p-values, and decreasing sample sizes and experiment durations. CUPED works by utilizing users' past behavior to adjust their metrics during experiments, thus explaining away some variance and correcting pre-exposure bias. The implementation has shown that around 9% of experiments gained statistical significance post-CUPED application, while 2% saw a decrease in significance, indicating a reduction in false positives. As companies grow and track metrics more consistently, the efficacy of CUPED is expected to increase, especially for experiments involving stable metrics over time. Statsig's rollout includes a new results card to display CUPED-adjusted metrics, encouraging users to explore this tool for more confident and expedited experimentation.