December 2025 Summaries

Trusting experiment results is crucial when deciding which feature variation to implement, but many experiments suffer from covariate imbalance, where baseline characteristics are unevenly distributed between control and treatment groups, leading to biased outcomes. This imbalance is particularly problematic in randomized experiments with small sample sizes or in B2B environments. For instance, an e-commerce platform might incorrectly conclude that a new checkout flow is ineffective if a few large enterprise customers skew the control group's average revenue. Stratified sampling addresses this issue by ensuring important user attributes are evenly distributed across groups, thus preventing random imbalances from affecting results. It involves defining what balanced means, evaluating candidate randomizations, and selecting the most balanced one, allowing for more reliable data analysis. This technique is especially useful when the user base is small or skewed, certain attributes significantly impact metrics, or there's a need for greater confidence in results before rollouts. LaunchDarkly offers a way to implement stratified sampling by using a CSV file of user data to assign experiment traffic, although it applies only to known users at test creation. By eliminating covariate imbalance, stratified sampling enables faster iteration, fewer false readings, and more informed decision-making.

LaunchDarkly has introduced a refined navigation update to its platform following feedback from users who found the previous design overwhelming due to the proliferation of items and icons. This update aims to enhance user experience by implementing collapsible sections, simplifying visuals with fewer icons and improved spacing, and elevating shortcuts for quicker access to frequently visited features. The improvements also include a streamlined Create action and an enhanced search function that is more keyboard-friendly, making it easier for users to navigate and stay focused on their tasks. These changes, building on a major redesign introduced a year and a half ago, are designed to reduce visual clutter, offer greater flexibility, and improve overall efficiency for all LaunchDarkly users, who are encouraged to share their feedback on the new experience.

LaunchDarkly has expanded its Data Export capabilities to support major data warehouses including BigQuery, Databricks, Redshift, and Snowflake, with new EU-region support, allowing teams to analyze feature releases and experiment outcomes directly within their data warehouses. This service enables the export of flag data, experiment metadata, and metric data, facilitating deeper analysis using trusted tools and sharing of insights across teams. By integrating with these platforms, teams can gain insights into user engagement, conversion rates, revenue impacts, and technical performance metrics such as page load times and error rates. Data Export is available for Enterprise and Data Export SKU customers, including those in the EU, and can be configured through the LaunchDarkly interface.

Teams integrating AI into their workflows face challenges of rapid iteration and risk management, with every model or prompt adjustment potentially impacting performance. LaunchDarkly and AWS provide solutions to facilitate safe and efficient AI deployments. Using AI Configs and Amazon Bedrock, teams can manage and test models and prompts in real time, allowing for dynamic parameter adjustments without redeployment. Bedrock Guardrails ensure safe user interactions by monitoring for sensitive content and policy violations, while LaunchDarkly's progressive rollouts and instant rollbacks provide controlled testing environments. These tools enable real-time evaluation of performance metrics and facilitate instant rollbacks if metrics decline, all while maintaining live applications. The integration of Amazon Q with LaunchDarkly further streamlines AI development by allowing developers to modify AI configurations using natural language, ensuring updates follow proper approval processes. This collaborative approach ultimately accelerates the adoption of new AI models and ensures that updates enhance both performance and reliability.

AI Configs introduces a novel approach to measuring the quality of AI systems in real-time through online evaluations, addressing the challenges posed by the nuanced nature of AI system behavior, which traditional software testing methods cannot adequately capture. By integrating this capability into the same control plane used for managing releases and experiments, AI Configs allows teams to continuously monitor and assess the performance of AI systems using metrics such as accuracy, relevancy, and toxicity. This real-time evaluation is facilitated by LLM-as-a-Judge, which automatically scores AI outputs to ensure quality standards are met and to guide decision-making during rollouts and experiments. AI Configs enables teams to make evidence-based decisions by comparing configuration variants and setting quality thresholds that trigger automatic adjustments if necessary. This system transforms quality measurement from a reactive process into a proactive and ongoing feedback loop, enhancing the ability to maintain high standards in AI performance and user experience. Currently available in early access, AI Configs offers tools for quality measurement that allow teams to address issues like tone drift and context loss, fostering a continuous learning environment to optimize AI outputs.

LaunchDarkly has introduced a new feature called Flag Audiences, which enhances the ability to trace and analyze feature flag evaluations by linking them directly to the specific users and sessions that experienced them. This feature allows users to view all evaluations for a flag, switch between various contexts, and filter by different variations, offering a comprehensive and actionable overview of rollout audiences. By integrating with the Observability SDK, Flag Audiences also provides the capability to watch session replays, enabling teams to understand user experiences in detail without leaving the platform. This real-time traceability is designed to improve incident resolution by swiftly identifying impacted users or sessions and understanding the context of their interactions, thus facilitating quicker debugging and collaboration among teams. Available to all LaunchDarkly Guardian customers, this addition aims to provide a unified view that combines flag evaluations, user sessions, and observability data, enhancing overall visibility and reducing the need for manual log cross-referencing.