December 2022 Summaries

In 2022, Statsig focused on enhancing its core product and expanding its suite of experimentation tools, seeing significant growth and adoption among its customers. The company's commitment to fostering a culture of experimentation led to securing Series B funding, spearheaded by Sequoia Capital, to continue developing innovative tools. Statsig introduced features that promote hypothesis communication, collaboration, and data organization, such as in-product discussions, tagging, and dashboards, to track and understand experimentation metrics. Additionally, the platform expanded its technical capabilities by integrating with data warehouses and releasing SDKs for various programming environments. Statsig's improved statistical engine now includes features like CUPED and Sequential testing to reduce bias, while a revamped power analysis calculator enhances precision in data handling. The company emphasizes user-friendly interfaces to make complex statistical methods accessible, aiming to support users in deriving meaningful insights from their experimentation efforts. Statsig's new blog and active Slack channel provide platforms for ongoing updates and community engagement as the company continues to innovate in the field of experimentation.

Astronomer hosted an in-person event at their new office, gathering data science and engineering professionals from companies like Google and BNY Mellon despite the busy holiday season. Organized by NYC Data Council's Keira Zhou from Spotify, the event featured presentations including Statsig's Vineeth Madhusudanan on data infrastructure for product experimentation and Astronomer's Viraj Parekh demonstrating Apache Airflow, both sparking lively audience engagement. Networking opportunities were plentiful, complemented by pizza and drinks, but the event was not livestreamed to emphasize face-to-face interaction. Photographs captured the event's energy, and there were discussions on topics like CUPED, experimentation culture, and A/B testing, with insights from industry experts like Ronny Kohavi and Allon Korem.

In a podcast episode titled "Minimum Viable Experimentation," Statsig founder Vijaye Raji and Sean Taylor engage in a discussion with hosts Tristan and Julia from dbt Labs’ The Analytics Engineering Podcast, exploring the significance of experimentation in scalable organizations. They emphasize the importance of empowering data scientists with the right tools to focus on strategic decision-making and experimentation rather than mundane tasks, highlighting that distributed decision-making and trust are key business outcomes. The conversation underscores the necessity of a company culture that embraces the potential of shutting down unproductive projects and values learning from both successful and failed experiments. Notable insights include the minimal success rate of experiments, yet the inherent learning value each provides, alongside reflections on experimentation strategies such as CUPED and notable shifts in product strategies influenced by A/B testing, as seen in companies like Facebook and Optimizely. The episode also delves into building a robust experimentation culture and infrastructure, drawing insights from experts in the field.

Vercel, a platform designed for developing, previewing, and shipping modern web experiences, offers seamless vertical integration from developer experience to edge delivery through advanced development tools and a serverless cloud environment. In partnership with Statsig, a leading experimentation and feature management platform, Vercel has introduced Edge Config, an ultra-low latency data store that allows for real-time feature flag and experimentation configuration updates. This integration enables Vercel users to access Statsig's server SDK at the edge, facilitating rapid feature management without network requests, resulting in reduced latency, improved reliability, and lowered infrastructure costs. The collaboration enhances end-user experiences by enabling faster website load times and decreased cumulative layout shift, with no additional cost for users who integrate the platforms. Additionally, the text touches on the evolution of web experience platforms, significant experimentation practices, and insights from industry leaders like Ronny Kohavi and Allon Korem, highlighting the importance of a robust experimentation culture and infrastructure.

Statsig provides support for SDKs across various languages and frameworks, including popular client-side technologies like React and JavaScript, and server-side languages such as Node.js and Python. These SDKs are designed to handle either single-user scenarios on the client side, where evaluations are done upfront, or multi-user server-side scenarios, where gates and experiments are evaluated locally. To ensure consistency across its SDKs, Statsig developed a server SDK consistency test that evaluates the behavior of SDKs against the Statsig backend. To streamline testing, they introduced Kong, a Typescript-based framework that allows for a "write once, run anywhere" approach, enabling comprehensive testing of SDK behavior using the Jest framework. These tests are crucial when making server or SDK changes to ensure stability and reliability. Future plans include extending the tests to long-running webserver scenarios to check for performance issues like memory leaks and race conditions, incorporating complex test cases, and using production traffic to validate SDK functionality. Additionally, insights into experimentation culture and strategies for running A/B tests effectively are discussed, alongside highlights of Statsig's dynamic work environment and innovative projects.

Unleash and Statsig have partnered to enhance feature management and product observability by integrating their platforms, allowing teams to conduct A/B testing with ease. Unleash, an open-source feature management tool, enables smaller, more efficient code releases, while Statsig offers data analytics and insights into feature performance and business metrics. Together, they facilitate the transition of partial rollouts into structured experiments, providing insights into the impact of new features on key performance indicators (KPIs). This integration automates data analysis, enabling teams to review feature performance before full deployment, and allows for the logging of various analytics events to track metric improvements across features. Users can join the Statsig x Unleash Developer Hangout and access resources like GitHub documentation and community support to further explore this collaboration.