June 2021 Summaries

Experiments+ is Statsig's comprehensive A/B/n testing tool designed to streamline the experimental process by allowing users to set up hypotheses, define metrics, and target completion dates while managing audience targeting and deploying multiple test groups to avoid experimental collisions. It offers a detailed reporting feature that evaluates experiments against key metrics using the "Pulse" view, helping users understand the impact on the entire suite of company metrics. Statsig promotes a simplified 3-step creation flow for Experiments+ to make A/B testing accessible and effective, with plans for further enhancements and features. The platform reflects a broader industry evolution in experimentation, influenced by insights from experts like Ronny Kohavi and Allon Korem, and draws from historical changes in testing practices that have shaped product strategies at companies like Facebook. Users can explore Statsig's offerings through a free account or demo, engaging with its SDK and console to start building experiments.

The text explores the concept of Product-led Growth (PLG), emphasizing the pivotal role of experimentation in driving sustainable business growth. It highlights how successful companies, such as Ford, Walmart, and major tech firms, have leveraged experimentation to refine their products and strategies, leading to significant market impact. The discussion underscores that while PLG focuses on using the product itself to engage and retain customers, the real "secret sauce" lies in the iterative process of testing and learning from user behavior. By doing so, businesses can evolve their offerings based on data rather than assumptions, facilitating faster and more informed decision-making. The piece also touches on the importance of a robust experimentation culture and infrastructure, as seen in companies like Facebook and Amazon, which allows for rapid testing and adjustment to market demands.

The text provides an overview of how Statsig's SDKs facilitate the evaluation of Feature Gates, commonly known as feature flags, in both single-user and multi-user environments. It explains that for single-user environments, Feature Gates and Dynamic Configs are fetched and cached upfront, while in multi-user server-side environments, initial implementation required an API request for each feature check, which proved inefficient. With the release of v3.0.0 of the node.js SDK, Statsig now allows for local evaluation of Feature Gate and Dynamic Config conditions, eliminating the need for frequent server requests and thus enhancing performance. The example of the 'is_statsig_employee' Feature Gate demonstrates how conditions are checked locally, and the text highlights the expansion of this functionality to SDKs in other programming languages like .NET, Java, and Ruby. Additionally, it touches on broader themes such as the importance of rapid experimentation and learning from A/B testing failures, as well as the evolution of web experience platforms and the role of culture in fostering innovation at Statsig.

A/B testing, also known as split or bucket testing, is a scientific method used to measure causality by comparing two groups: a test group receiving a new feature and a control group with the base version. This method, which is widely adopted in clinical trials and digital marketing, helps organizations like Facebook, Netflix, and Amazon optimize products by determining the impact of changes on user engagement metrics, while controlling for confounding factors through randomization. The process involves statistical testing to identify if differences between groups are statistically significant, with results considered significant if they exceed a set threshold, usually 95%. A/B testing offers a comprehensive view by evaluating primary, secondary, and ecosystem effects, allowing businesses to make informed decisions, and is supported by platforms like Statsig, which makes the process accessible to all levels of experimenters.

The article details the journey of a designer joining Statsig, an early-stage startup, to establish a cohesive design system. Initially relying on Google's Material Design Component Library, the team prioritized product development over design consistency. The designer opted to create a new design system after identifying recurring patterns and inconsistencies, leveraging experience gained at Facebook. Despite the risks associated with early-stage startups, the decision was driven by the potential for improved UI/UX consistency and efficient design-engineering collaboration. The newly implemented design system, facilitated by tools like Figma, has enhanced product development speed and effectiveness, fostering a culture that encourages adaptability and innovation. Statsig, founded in 2021 with a mission to help teams utilize data for better products, benefits from a team of ex-Facebook employees replicating successful internal tools to empower product teams. The article also touches on experimentation insights from industry experts and the evolution of A/B testing practices.

Kirkland, known for being the birthplace of Costco and its Kirkland Signature brand, hosts the office of Statsig, which was established with a focus on fostering organic product evolution through in-person collaboration. The Statsig office was chosen for its central location, modern facilities, and alignment with the company's values of openness and transparency, featuring an open floor plan and ample white-boarding space. Situated in a region with a thriving startup ecosystem and proximity to tech giants like Amazon, Microsoft, Google, and Facebook, the office offers a dynamic environment for team collaboration. Statsig emphasizes the importance of A/B testing and experimentation culture, drawing insights from industry experts like Ronny Kohavi and Allon Korem to drive innovation and learning from failures. The vibrant office culture is characterized by hackathons, team-building activities, and exciting projects, contributing to its unique and engaging work atmosphere.