April 2025 Summaries

Statsig Release Pipelines offer a new method for managing Infrastructure as Code (IaC) through multi-stage rollouts, ensuring infrastructure changes are executed as safely and gradually as application code updates. By integrating with Statsig feature flags, Release Pipelines allow platform engineers, SREs, and DevOps teams to conduct phased infrastructure changes across different regions or environments with added safeguards, reducing risk and improving feedback loops. The system works in tandem with the IaC tool Pulumi, where Statsig orchestrates the staged rollout and Pulumi executes the necessary infrastructure changes. This setup allows for real-time control over the release process, offering commands for manual approvals, pauses, or rollbacks as needed. A demo using a simple service called ColorTeller illustrates how the pipeline enables a gradual, region-by-region deployment, ensuring a smooth transition and verification of changes before a full rollout. Currently in beta, Statsig Release Pipelines aim to bring the advantages of progressive delivery to the infrastructure layer, promising faster and safer deployments.

At the recent Product Growth Forum, industry leaders gathered to discuss the complexities of building enduring products in an ever-changing world, focusing on the gradual and transformative impact of AI, the importance of context in growth strategies, and the power of saying "no" to prioritize effectively. Notable insights included the need for AI-native software to evolve slowly and messily, the significance of balancing product stability with experimentation, and the unconventional yet impactful growth strategies that sometimes involve subtractive efforts rather than additive ones. The forum also highlighted the role of marketing in the AI era, emphasizing the importance of brand identity over features, and addressed the tension between product-led growth (PLG) and traditional enterprise sales in B2B environments, noting that high-touch relationships are essential for securing significant deals. The event concluded with personal growth advice in the AI age, urging attendees to embrace continuous learning and adaptability, underlining that successful product development requires a mix of intuition, focus, and storytelling.

For companies engaged in frequent experimentation, as opposed to startups with occasional A/B tests, adopting a robust framework that emphasizes coverage, metric sophistication, and hypothesis quality is crucial. Full coverage ensures every feature undergoes testing, preventing blind spots and ensuring that only safe features are released. Sophisticated metrics transition from simple KPIs to an Overall Evaluation Criterion (OEC) that encompasses revenue, engagement, risk, and customer sentiment, necessitating cross-organizational collaboration for comprehensive insights. High-quality hypotheses evolve from isolated tests to cumulative learning, with each experiment informing future decisions, supported by a centralized knowledge base. While challenges such as balancing velocity with coverage, avoiding data overload, and maintaining disciplined curiosity exist, companies that manage complexity effectively turn experimentation into a continuous cycle of improvement, rather than treating tests as isolated gambles.

Statsig has introduced release pipelines as a new feature to enhance the safety and efficiency of deploying changes across complex infrastructures, complementing their existing feature flag tool. While traditional feature flags focus on user segmentation, they may overlook infrastructure-level challenges, particularly at high release velocities where even minor risks can significantly impact performance. Release pipelines enable multi-stage rollout strategies that take into account infrastructure boundaries, allowing changes to be rolled out environment by environment, targeting specific infrastructure segments, and progressing through stages with controlled intervals or approvals. This approach allows for early issue detection and prevents large-scale outages by containing potential failures within specific segments before they impact the broader system. The dual focus on user and infrastructure-level controls aims to provide engineering teams with greater flexibility and reduced overhead, ensuring changes are validated in real environments with minimal risk. Currently in beta, release pipelines are available for enterprise customers, encouraging collaboration within Statsig's community to foster an experimentation culture.

Statsig, a company managing over 24 SDKs with a small development team, has transitioned to a centralized Rust-based core engine to improve efficiency and performance across its server-side features. Previously, each SDK required duplicating evaluation logic across multiple languages, creating significant maintenance challenges. By consolidating the most popular server SDKs—Node, Python, Elixir, Java, and Rust—into a single Rust core, the company has achieved up to fivefold improvements in evaluation speeds, alongside enhanced code sustainability. Despite initial challenges and increased workload during the transition, the move aims to decouple developer resources from the number of supported languages, allowing for faster feature deployment and long-term maintenance simplification. The project utilized various Foreign Function Interfaces (FFIs) to create bindings for different languages, and the team acknowledges the supportive Rust community for aiding in the learning and implementation process. Moving forward, Statsig plans to focus new developments on the Server Core SDKs, maintaining legacy SDKs in a limited capacity, and anticipates that this strategic shift will streamline future updates and development efforts.

Over the past decade, the practice of experimentation has transformed from a marketing tactic to a fundamental strategy in software development, with tech giants like Amazon and Google leading the charge. This shift has been fueled by the rise of A/B testing, feature flagging, and various experimentation platforms, enabling companies to make data-driven decisions that enhance product development and innovation. Industry leaders such as Jeff Bezos have championed experimentation as a core component of their business strategy, emphasizing its role in fostering rapid innovation through continuous testing and learning. The proliferation of experimentation tools has democratized the process, allowing even small teams to conduct experiments and improve their decision-making. As a result, experimentation has become an integral skill for tech workers, driving career growth and company success. The practice is set to expand further, influencing areas like AI and offline domains, solidifying its status as a dominant strategy in the software industry and beyond.

Marketing attribution has evolved significantly from early models like Media Mix Modeling (MMM) to sophisticated data-driven approaches, addressing the growing complexity of customer journeys across digital channels. Initially, single-touch models such as last-click attribution were favored for their simplicity, but these often failed to account for the entire customer journey. As interactions became more multifaceted, multi-touch models emerged, albeit with limitations due to inherent biases and rigid rules. The advent of data-driven models, utilizing algorithmic techniques like Markov chains, Shapley values, and deep learning, has allowed for more nuanced attributions by analyzing actual conversion paths and capturing complex interactions. Meanwhile, incrementality testing and causal inference methods have gained traction as they strive to isolate the true impact of each channel. Although rule-based models offer clarity and ease of implementation, data-driven and experimental approaches promise deeper insights into channel effectiveness, with no single model universally deemed the best. Organizations often combine various methods and validate them through controlled experiments to achieve the most accurate insights, adapting to ongoing changes in privacy regulations and digital landscapes.

Statsig's quarterly hackathons are a vibrant tradition where employees set aside their regular duties to engage in creative, fast-paced projects that enhance the company's culture and innovation. These events bring together engineers, designers, marketers, and product managers to develop experimental features, automations, or even just-for-fun tools over two days, often resulting in prototypes or internal tools that may or may not be integrated into production. The Q1 2025 Hackathon featured a wide array of projects across AI and automation, analytics and experimentation, infrastructure and developer experience, as well as culture and branding. Highlights included an AI assistant for code management, an AI-powered setup wizard to simplify product integration, and various tools to enhance user experience in analytics and data visualization. Additionally, projects focused on company culture and visual identity, such as revamping the office's poster room and initiating a pop-up thrift shop for employees, underscored the event's role in fostering a creative and collaborative work environment. Overall, these hackathons are a testament to the innovation, creativity, and community spirit that define Statsig.

SEO A/B testing involves experimenting with changes to website content to measure their impact on search engine rankings, despite the challenges posed by an opaque algorithm and reindexing delays. Unlike typical A/B tests that use user-based randomization, SEO experiments require randomization by URL due to the necessity of measuring changes over a longer time scale. This method involves modifying some pages and observing their performance, though it can be complex due to the need for a sufficient number of pages and ensuring even distribution between test and control groups. Common SEO tests might include altering page titles or optimizing images, and these tests, while seemingly simple, can yield significant results. Tools like Statsig offer flexibility in randomization and support for various environments, helping manage potential biases and ensuring experiments are well-structured. The piece highlights the importance of using a robust experimentation platform to integrate SEO tests with other types of experiments, providing tools to handle challenges such as pre-experimental bias and uneven bucketing.

Statistical significance is a fundamental concept in data analysis that helps distinguish meaningful data patterns from random chance, thereby ensuring the reliability of analytical results. By testing for statistical significance, analysts can determine if observed effects or relationships are genuine, guiding informed, data-driven decisions and minimizing the risk of false positives. This involves formulating clear null and alternative hypotheses, selecting appropriate significance levels (α) to balance Type I and II error risks, and employing the right statistical tests based on data type and research questions. Techniques like multiple testing corrections, such as the Bonferroni and Benjamini-Hochberg procedures, help control for false discovery rates when conducting numerous tests. Understanding p-values is crucial; they indicate the probability of observing data as extreme as measured under the null hypothesis, with low p-values suggesting that the results are unlikely due to chance alone. Although statistical significance is vital, it does not necessarily imply practical relevance, and analysts must consider effect size and real-world implications. Sample size, power analysis, and controlling for confounding variables are critical to ensure robust and meaningful analysis, while avoiding practices like p-hacking preserves the integrity of the results.

Statsig Server Core is a performance-enhanced rewrite of various server SDKs, incorporating a shared Rust library that significantly improves evaluation speeds by up to five times. This update not only accelerates server-side metrics evaluation and processing but also introduces new features such as Parameter Stores, SDK Observability Interfaces, and streaming flag/experiment changes via the Statsig Forward Proxy. While there are no immediate plans to deprecate legacy SDKs, users are encouraged to transition to Server Core to benefit from these advancements. This shift allows Statsig to prioritize the development of meaningful features over SDK maintenance, aligning with their strategy to reduce technical debt and focus on long-term customer needs. For further inquiries or assistance with migration, Statsig offers support through their Slack channel.

A/B testing is crucial for data-driven teams, but its true value lies in enabling qualitative decisions that go beyond mere metrics. Real-world decision-making involves weighing diverse perspectives and objectives that cannot always be captured in quantitative terms. While statistically sound results are necessary, understanding the broader context, including brand considerations and alignment with business goals, is equally important. Data science faces challenges in inference and extrapolation, requiring a balance of quantitative rigor and qualitative judgment. Historically, significant innovations were achieved with domain expertise and instinct, emphasizing the importance of experience-driven interpretation alongside data. Successful data scientists blend numerical analysis with qualitative insights, framing decisions as trade-offs and inviting diverse feedback to inform outcomes. Ultimately, effective decision-making integrates robust data with thoughtful interpretation, making data scientists who master both quantitative and qualitative approaches invaluable to experiment-driven cultures.

Statsig has launched CURE, an advanced regression adjustment tool for experiments, which builds upon the existing CUPED framework by allowing the incorporation of arbitrary covariate data to reduce variance more effectively. CURE is especially beneficial for experiments involving new users or scenarios without historical data, enabling more precise measurement with fewer participants. By leveraging a variety of user attributes as covariates, such as country or behavior predictions, CURE aims to provide a transparent and flexible approach to regression adjustment without the complexity of a "black box" system. The tool supports project-level configuration and managed feature selection to maintain data quality and prevent issues like collinearity. Initial tests have shown a 10-40% decrease in experiment runtimes compared to CUPED, highlighting its potential for improving experiment efficiency and insight into metric behavior.

Implementing feature flags in serverless architectures such as AWS Lambda, Google Cloud Functions, and Cloudflare Workers presents challenges due to the transient nature of these environments, impacting cold starts, latency, and microservice dependencies. Feature flags allow developers to manage code without redeployment, but they introduce complexities like inconsistent flag states and potential tech debt. Three main solutions are proposed: using centralized feature flags with Statsig to maintain consistency and manage lifecycle; creating a custom flagging solution with external data stores like Cloudflare Workers KV, which, while potentially reducing latency, may introduce security risks and maintenance challenges; and integrating an external data store with a centralized platform like Statsig, which balances performance with maintainability by offloading costly network calls. Each solution offers distinct advantages and drawbacks, allowing developers to use feature flags effectively while maintaining agility and ensuring safe rollouts in serverless environments.

Statsig has implemented a series of enhancements to its dashboards aimed at improving speed, ease of navigation, and functionality without sacrificing performance. Key updates include cohort filtering, support for quick values in funnels, and the ability to use formulas within widgets, which collectively enhance data exploration workflows for product and growth teams. The introduction of just-in-time widget loading minimizes unnecessary data queries, resulting in faster load times and a smoother user experience, particularly for complex dashboards. Additional improvements involve more intuitive widget duplication, enhanced editing for text and pulse widgets, the ability to view widgets in fullscreen, and the addition of a new share button and refresh option for Warehouse Native dashboards. These updates, now available without requiring migration, also extend to the web analytics dashboard, which now benefits from quicker load times and improved data views. Feedback from users is welcomed through various communication channels, and Statsig offers demos to address any questions related to experimentation.

Statsig has launched a new product analytics solution that integrates seamlessly with Microsoft Fabric, aimed at simplifying the process of running large-scale product analytics. This integration allows teams to use a single source of truth for analytics, eliminating the need for multiple tools and complex data pipelines while maintaining security and compliance. By operating within the existing Fabric environment and utilizing data stored in OneLake, teams can quickly connect their data, define key metrics, and create analytics workflows such as segmentation, dashboards, and funnels. Statsig's solution offers advanced capabilities like A/B testing and feature rollouts without requiring SQL knowledge, making it accessible to a wide range of users. This approach offers an incremental path to analytics maturity, enabling teams to innovate and scale efficiently by leveraging the power and simplicity of Statsig's integration with Microsoft Fabric.

A/B testing analysts frequently face challenges with skewed distributions and outliers in Key Performance Indicators (KPIs), particularly with revenue metrics, which can distort test results by inflating variance and reducing statistical power. This piece explores the issues caused by outliers, highlighting the increased probability of Type II errors in hypothesis testing, where real effects might be missed. Methods for identifying outliers include visual tools like boxplots and statistical techniques such as Z-scores and Interquartile Range (IQR). Handling outliers requires careful strategies, including data transformations and winsorization—the latter being recommended for preserving data integrity while minimizing variance impact. Winsorization involves capping extreme values at predefined percentiles to maintain statistical power and enhance the reliability of test outcomes. The article underscores the importance of managing outliers to improve the sensitivity of A/B tests, as demonstrated through simulations showing increased statistical power with winsorized datasets.

Statsig has introduced the Single Pane of Glass™, a groundbreaking collaboration tool designed to unite teams by providing a comprehensive view of product strategies and facilitating real-time discussions and decisions. This minimalist, seamless integration layer is compatible with existing tools like Jira, Figma, and Slack, offering features such as zero latency, no login requirements, and support for unlimited users in proximity. Recognized as a leader in the 2025 Magic Quadrant for Glass-as-a-Service (GlaaS) Platforms, the Single Pane of Glass aims to break down silos and enhance team collaboration across companies of all sizes.