October 2025 Summaries

Earlier this year, Statsig launched Statsig Server Core, a revamped version of their Server SDKs using a unified Rust evaluation engine to optimize performance across various platforms. The focus was on reducing memory usage, and through rigorous testing, significant improvements were made compared to the legacy SDKs. Initial versions of Server Core exhibited higher memory consumption and volatility, but after optimizations, the latest version showed an 84% reduction in baseline and a 68% reduction in peak memory usage. Key enhancements included string interning, shared DynamicReturnable objects, and streaming deserialization, which collectively minimized memory spikes. These optimizations are particularly impactful as they apply universally across all SDKs backed by the Rust core, ensuring a leaner, more scalable, and cost-efficient performance for users. The results validate the shift from language-specific SDKs to a unified core, demonstrating a smaller memory footprint, faster operations, and easier maintenance across different environments.

A/B testing frequently encounters the challenge of multiple comparisons, which can inflate false positive rates when conducting multiple hypothesis tests simultaneously. To address this, statistical correction methods such as the Bonferroni correction, Dunnett’s test, and the Benjamini-Hochberg (BH) procedure are employed, each with distinct approaches to controlling error rates. The Bonferroni correction, known for its simplicity, is highly conservative, reducing statistical power by adjusting the significance threshold for each test to maintain the family-wise error rate (FWER) below a specified level. Dunnett’s test, more powerful than Bonferroni in specific contexts, compares multiple treatment groups to a single control group, adjusting for dependencies between hypotheses. The BH procedure, focusing on the false discovery rate (FDR), offers a balance between controlling false positives and maintaining power, making it suitable for exploratory studies and scenarios with many tests. In cases of sequential testing, often arising from peeking at accumulating data, methods like alpha spending functions or the Mixture Sequential Probability Ratio Test (mSPRT) are used to manage the increased risk of Type I errors. Selecting the appropriate method depends on the research context, the importance of controlling false positives, and the need to preserve statistical power.

Statsig's recent out-of-home advertising campaign at Austin–Bergstrom Airport creatively capitalized on the consecutive F1 Grand Prix and EXL 2025 events to showcase its brand. A small, agile Brand and Marketing team executed the campaign in just two days, featuring two distinct segments that maintained the core message of helping product teams move faster. The F1 segment used AI-generated visuals and playful language to capture the excitement of the race, while the EXL segment focused on clarity and precision to appeal to a crowd of product and experimentation leaders. This campaign demonstrated Statsig's ability to adapt its brand identity to different themes while maintaining a consistent message, highlighting both creative agility and effective cross-functional collaboration.

In an effort to enhance their creative process for an in-house out-of-home (OOH) campaign, a team experimented with integrating AI tools like ChatGPT-5 and Sora to accelerate concept development and execution. Initially hesitant about the potential low quality and originality of AI-generated images, they found that AI significantly increased their creative velocity by quickly generating visuals, creating copy variations, and researching contextually relevant information. The creative process involved an early phase of brainstorming and idea generation, followed by a mid-phase of refining visual directions using AI prompts tailored for different models. Despite the speed and efficiency AI brought to these phases, limitations in image generation led to frustration in the execution phase, requiring extensive post-editing to correct AI's visual inconsistencies. The campaign successfully illustrated the potential of AI to speed up early creative stages, while also highlighting the importance of precise creative direction and the possibility of future investments in more advanced resources.