August 2026 Summaries

LaunchDarkly describes its effort to adapt to agent-driven software development amid rapid technological change, acknowledging that even an AI-forward company has experienced uncertainty, confusion, and challenges in managing organizational change. The company is building an “AI software factory” that integrates agents into its software delivery lifecycle to reduce manual work, such as determining feature-flag placement, identifying existing flags, and measuring outcomes. Initially called Project Fairytale, the initiative developed through parallel research into human workflows and cautious deployment of agents that could contribute real code, later expanding into prototypes with design partners. Alongside technical experimentation, LaunchDarkly emphasizes the cultural need to share lessons and failures openly, accept discomfort and uncertainty, collaboratively define emerging practices, and ensure that adoption of AI advances with—not at the expense of—the people involved.

A LaunchDarkly engineer describes using an internal AI software factory to build and operate a personal AI baseball analytics application, testing how automated feature-flag workflows affect software delivery. The Docker-based application combines baseball datasets with AI models to support chat, reports, player analysis, game replay, and pitch-sequencing recommendations across local, home-server, and public cloud deployments. Its roughly 50 feature flags govern feature availability, destructive operations, authentication, access controls, runtime model prompts, and interface layouts, while the factory automatically creates flag wiring, metrics, tests, and in-app status information during pull requests. The approach enabled a rapid rollback after a redesigned interface caused a blank-page bug and reduced risks around bulk deletion, public access, and authentication rollout. The author also notes operational challenges, including remembering to activate dark-by-default flags after deployment, ensuring clients support both enabled and disabled paths, and resolving mismatches between configured flags and code integration.

LaunchDarkly’s latest updates focus on bringing monitoring, remediation, AI-assisted maintenance, and experimentation closer to the point of software release. Adaptive triggers can automatically change a feature flag when observability signals such as error rates cross configured thresholds, while Session Replay connects flagged audiences with recordings, logs, traces, and flag evaluations to aid investigation. Vega Flag Cleanup uses an AI agent to identify stale flags, make code changes, and open reviewable pull requests, with scheduling and safeguards for critical environments; MCP integration also allows agents such as Claude, Cursor, and Codex to access experimentation and observability data and initiate triage from tools like Slack or PagerDuty. Warehouse-native Experimentation keeps analytics and experiment metrics in a company’s existing data warehouse, supporting Snowflake, BigQuery, Databricks, and Redshift, while allowing teams to add metrics or segmentation attributes during active experiments without restarting them.

LaunchDarkly is now available through the Vercel Marketplace, allowing developers to create or connect a LaunchDarkly account, provision projects and environment-specific SDK keys, and begin using feature flags within Vercel-managed workflows. The integration mirrors Vercel’s Development, Preview, and Production environments, syncs keys as environment variables, supports Vercel single sign-on, and keeps flags and projects visible in the Vercel dashboard while allowing access to the full LaunchDarkly application. It also supports Vercel Global Config, enabling flag evaluation in Middleware and Vercel Functions without network calls to LaunchDarkly. Billing is handled through Vercel at the same pricing as direct LaunchDarkly plans, beginning with a free Developer plan, while users can expand into progressive rollouts, automated rollback, experimentation, and AI configuration management.

Containerized AI applications combine Docker and Kubernetes with MLOps practices, but their large model artifacts, GPU scheduling requirements, scaling behavior, and separation of batch training from continuously served inference make them more complex than conventional services. Traditional CI/CD systems such as Jenkins, GitLab CI/CD, and GitHub Actions remain useful for building, testing, and deploying containers and infrastructure, yet they are slower and less suitable for frequent changes to prompts, model choices, inference parameters, and provider routing. The proposed approach separates code and infrastructure releases from dynamic application configuration through feature-management tooling such as LaunchDarkly AgentControl Configs, allowing running services to receive prompt, parameter, and model updates without redeployment while preserving version histories and audit trails. It also supports progressive percentage rollouts, AI-specific quality monitoring for measures such as relevance, hallucinations, costs, and latency, and automated rollback when evaluation scores decline. To reduce dependence on individual model vendors, the architecture can abstract multiple providers and dynamically fail over among primary, cloud backup, or local options in response to outages, rate limits, compliance needs, or regional performance conditions. The overall framework emphasizes controlled experimentation and production measurement to balance response quality, cost, latency, and user satisfaction while retaining established CI/CD workflows for substantive application and infrastructure changes.

Effective ML and LLM experiment tracking extends beyond hyperparameters to include data versions, code commits, environments, model artifacts, prompt templates, tokenizer and sampling settings, evaluation results, resource usage, and the reasons runs were triggered. It distinguishes experiment tracking at the individual run level, model tracking across deployment stages, data tracking for training inputs and lineage, and prompt tracking for LLM inference behavior, arguing that gaps among these records make production regressions difficult to reproduce or diagnose. A scalable tracking system should use durable metadata and artifact storage, automated CI/CD logging for successful and failed runs, evaluation gates linked to model registries, and production monitoring that feeds drift, latency, cost, and quality signals back into triage or retraining. LaunchDarkly AgentControl is presented as a runtime configuration layer that versions prompts, model parameters, and tools, connects offline validation to live configurations, and supports controlled rollouts, targeting, A/B testing, model switching, and rapid rollback without redeployment. Together, these practices create an auditable chain from an experiment and its inputs to the configuration served to users, enabling teams to release changes more safely and investigate incidents more quickly.

Experiment tracking is presented as essential infrastructure for production machine learning and LLM systems, providing a centralized, reproducible record of each training run’s configurations, metrics, artifacts, code version, execution environment, data and feature lineage, resource use, and links to model registry and deployment stages. Effective systems support scalable storage, distributed training, CI/CD automation, access controls, audit trails, run comparison, and integration with monitoring so teams can evaluate candidates, detect regressions, retrain models, and meet governance requirements. LLM workflows require additional tracking of prompts, sampling parameters, tokenizer and base-model versions, fine-tuning artifacts, quality evaluations, and substantial compute costs. The text distinguishes experiment qualification from runtime deployment control, describing model registries and tools such as LaunchDarkly as mechanisms for gradual exposure, monitored rollouts, and rapid rollback. It also warns against local-only records, missing lineage, overwritten runs, incomplete metrics, absent environment details, and disconnected registries, while noting that lightweight tracking may be sufficient for purely temporary exploration that will not affect deployment or long-term decisions.

Experiment tracking is presented as essential infrastructure for production machine learning and LLM systems, replacing fragmented records with a reproducible history of each training run’s configurations, metrics, artifacts, code version, execution environment, dataset and feature lineage, resource use, and downstream model-registry status. Effective systems centralize metadata through resilient, access-controlled tracking servers, use scalable storage for metrics and large artifacts, integrate with version control, feature stores, CI/CD pipelines, monitoring, and model registries, and support comparison, search, distributed logging, cost visibility, and governance. For LLMs, tracking must additionally capture prompts, sampling settings, base models, tokenizer versions, fine-tuning artifacts, evaluation rubrics, and serving quality and cost signals. Automated evaluation gates can qualify candidates for registration, while runtime controls such as LaunchDarkly feature flags and AgentControl can govern gradual exposure, monitor live performance, and roll back problematic versions without redeployment. The discussion emphasizes that lineage, immutable records, environment capture, and audit logs are especially important for regulatory compliance and troubleshooting, warns against local-only storage, missing data references, overwritten runs, and incomplete metrics, and notes that lightweight tracking may be sufficient for disposable exploration but becomes necessary when models affect users, business decisions, safety, or compliance.

A LaunchDarkly engineer describes how an apparently routine frontend migration from an older API to a newer one introduced production errors, but a feature flag and automated guarded release limited exposure and rolled back the change after 13 of 243 affected users encountered errors while none of 250 control users did. Using a release dashboard screenshot, Claude queried Datadog and quickly identified that the new backend endpoint was incorrectly applying an entitlement check that the old endpoint lacked, causing authorization failures for some users. After the check was removed from the read path, the release was restarted successfully. The experience illustrates LaunchDarkly’s AI-powered software factory approach, in which runtime controls, progressive traffic rollout, metric monitoring, and automatic rollback help make deployments safer, alongside internal efforts to automate flag creation, guarded releases, and later cleanup.

Marek Poliks of LaunchDarkly argues that while AI gateways are important centralized access-control points, especially for mature enterprises, they can create latency, external dependencies, single points of failure, and privacy risks because they may process sensitive prompts and model responses. He contends that gateways have limited visibility into an agent’s full runtime harness, including its available tools and skills, which restricts their ability to enforce meaningful governance beyond basic model access and failover policies. Poliks favors application-level runtime controls that can apply guardrails and online evaluations near the agent itself without transmitting customer context or adding mid-execution vendor API calls. He also maintains that observability, logs, and traces remain valuable but are inherently retrospective, making active intervention and prevention during execution essential as agents gain access to more consequential systems and information.

Agent Optimization, currently in beta within AgentControl, automates the process of improving agent configurations based on user-defined criteria such as quality, cost, and speed, without being tied to a specific framework. Users outline what constitutes a good response and select models for testing, while the optimization loop generates and evaluates candidate configurations using a large language model (LLM) judge. This process alleviates the traditionally labor-intensive task of manually iterating on agent improvements, as it allows for automated generation, scoring, and comparison of variations against a baseline, with the option to optimize further for cost and speed. Users define the acceptance criteria and configuration limits, and the system handles the iterative adjustments, scoring, and feedback loop to ensure that any new configuration surpasses the current setup. The tool supports different modes for optimizing known behaviors or exploring new patterns, enabling continuous refinement in response to evolving inputs or model updates. By offloading the repetitive work to the automation process, teams can focus on defining what 'better' means for their specific use case, while the system iterates through potential configurations to find an optimal solution that maintains the established quality bar.

LaunchDarkly engineers embarked on a project to modernize their most complex and business-critical frontend codebase using AI-driven software factory patterns, challenging the ideal of a fully autonomous "dark factory." Initially, the plan was to rewrite 66,000 lines of React code within six weeks, but the reality proved more complex, requiring human oversight to effectively manage the process. The team discovered that while agents can handle scale, human intervention is crucial for steering and improving outcomes, leading them to break the project into manageable phases. Leveraging feature flags and controlled automation, they safely integrated new code, ensuring zero disruption to customers while maintaining functional and visual parity. The experience underscored the importance of human judgment in AI-driven development and highlighted the potential for ambitious projects to be tackled by small teams with the right infrastructure. By focusing on the most challenging parts of their codebase, LaunchDarkly demonstrated the viability and benefits of a balanced AI software factory approach.

LaunchDarkly engineers have developed a system to improve the AI software development lifecycle by introducing a tool called "vent" within their MCP toolset, which enables agents to report issues such as bugs or missing parameters in real-time, thereby creating a closed-loop feedback system. When agents encounter a problem, the vent tool collects and triages the feedback, triggering a series of automated processes that include triaging the issue, notifying team members via Slack, creating Jira tickets, and deciding whether to fix or escalate the issue. This process has transformed agent feedback into actionable improvements, resulting in over 100 triaged tickets and pull requests. To enhance the reliability of fixes, the agent environment has been taught to perform quality assurance similarly to a human, using automation to validate fixes against real API responses. This approach not only accelerates tool improvement by capturing issues promptly but also offers insight into an automated software factory where feedback and remediation are seamlessly integrated, reducing busywork and filling in critical gaps. Inspired by Lovable, LaunchDarkly invites users to join a waitlist for early access to these developments.