May 2021 Summaries

Founded in February 2021 and based in Kirkland, WA, Statsig is a company dedicated to empowering product teams to use data effectively in order to build better products, inspired by the growth infrastructure from Facebook. The company, staffed by former Facebook employees, offers tools for data analysis, visualization, and interpretation, allowing teams to make informed decisions about feature launches. Recently, Statsig has revamped its brand identity to better communicate its mission as a reliable, data-driven service, with a new logo designed to reflect modernity and user-friendliness, traits often missing from other B2B SaaS products. The logo's design elements were carefully chosen to represent the company's strategic focus, and it is being integrated across all customer touchpoints, including social media and the company's website. As Statsig continues to grow, it is also fostering a strong internal culture through initiatives like hackathons and team-building activities. The company plans to publish its brand guidelines soon, amid ongoing efforts to establish a strong experimentation culture and infrastructure, drawing insights from industry experts like Ronny Kohavi and Allon Korem.

At Statsig, the process of measuring the cumulative impact of multiple feature launches is enhanced through the use of "Holdouts," a method inspired by practices at Facebook. Holdouts involve withholding a small percentage of users from experiencing new features for a set period, providing a baseline to assess the collective effects of these features over time. This method is particularly useful for evaluating long-term impacts, detecting subtle changes, and analyzing metric shifts. Statsig offers a user-friendly setup for implementing holdouts, with features like automatic impact computation via the Pulse engine, and options for global application or back-testing existing features. The platform also supports a free plan to encourage experimentation without sales intermediaries. Additionally, the text discusses the significance of CUPED for running efficient experiments, insights from industry experts on fostering an experimentation culture, and reflections on transformative experiences at companies like Optimizely and Facebook, highlighting Statsig's unique culture and dedication to innovation in A/B testing.

Statsig employs "Feature Gates" as a mechanism to control the release and testing of new software features by acting as gatekeepers that determine user eligibility based on predefined rules and conditions. These gates allow developers to manage feature rollouts by specifying conditions, such as user email domains, and percentages of users who can access features, facilitating gradual releases and A/B testing. The process involves creating rules with conditions, which are evaluated to decide if a user should have access to a feature. This approach helps decouple development cycles from release cycles, enabling teams to validate features and measure impacts before broader deployment. Statsig offers tools to enhance development workflows and experimentation, encouraging developers to explore these capabilities through their free services and demo accounts.

The author describes their transition from using Objective-C to Swift for iOS development at Facebook, highlighting Swift's more accessible syntax and useful features like optional parameters and true generics, which facilitate cleaner code and user-friendly function signatures. They explain how Swift's syntax, resembling languages like JavaScript or Java, makes it easier for new developers to understand, and provide examples of how Swift simplifies tasks like string concatenation and handling optional values. The author also discusses the interoperability between Swift and Objective-C, detailing how Swift can be used alongside Objective-C by creating wrappers for Swift classes and functions. Despite the ease of implementation, the author notes a lack of comprehensive documentation for this process and encourages readers to explore their GitHub repository for more insights. The text concludes by alluding to broader industry topics such as experimentation in A/B testing and the evolution of platforms like Optimizely, suggesting that the author's experiences at Statsig involved engaging projects and a unique company culture.

RUIDs (Rodrigo’s Unique Identifiers) are a system for generating 64-bit unique identifiers designed to be mathematically unique within the same RUID root, which is a set of generators identifiable through shared configuration. The system uses 41 bits for the timestamp, 14 bits for a sequence number, and 9 bits for the root id, further divided into cluster and node ids. RUIDs accommodate time travel by using a millisecond maximum time travel threshold, although this design has limitations when the generator is inactive during time travel. Written in Rust and highly performant, RUIDs can be integrated easily via an actix HTTP server, and while primarily developed for Rodrigo’s benchmarking needs against 128-bit UUIDs, they offer a simpler alternative to existing systems like Instagram's IDs, Twitter’s Snowflake, and Sony’s Sonyflake, without external dependencies. Despite their specific use case, RUIDs can be ported to other environments due to their minimalistic codebase, and they are documented on GitHub for those interested in implementation details.

The text discusses the concept and implications of multi-arm experiments, also known as multi-group or A/B/n testing, which allow companies to test multiple hypotheses simultaneously by comparing several test groups against a control group. This method is efficient in terms of sample size, cost, and time but carries the risk of Type I errors (false positives), which can occur due to statistical randomness. To mitigate this, statisticians often use the Bonferroni correction to adjust the significance level, reducing the chance of false positives at the cost of increasing Type II errors (false negatives). The author suggests considering the trade-offs between these errors, especially in scenarios where there is confidence that the control group is suboptimal, and advises maintaining a significance level of ⍺=0.05 for up to four variations in a multi-arm experiment. The text underscores the importance of understanding these trade-offs to make informed decisions in product development and experimentation, and it concludes by recommending Statsig as a tool for running such experiments.