January 2019 Summaries

DevOps is a cultural shift that emphasizes collaboration between development and operations teams, enabling faster delivery of software through Agile methodologies and Continuous Integration/Continuous Delivery (CI/CD) practices. This approach involves smaller, more frequent releases with the understanding that they may not be perfect and may require updates. Feature management platforms have emerged to support DevOps workflows by providing control points and safety valves during the release process. These platforms allow teams to separate code deployment from feature release, enabling greater control over user experiences and faster feedback loops. By empowering cross-functional collaboration, DevOps helps organizations innovate faster and deliver higher value to businesses and customers.

For an experimentation program to yield tangible business results, understanding what to test, how to test it, and how to analyze results is crucial. Excessive or misguided testing can waste resources and undermine stakeholder confidence. The aim should be to conduct strategic experiments that deliver cumulative marginal wins, focusing on when to experiment, choosing appropriate tools, and translating outcomes into actionable insights. High-impact experiments are characterized by high uncertainty, conflicting opinions, random audience exposure, and measurable outcomes. It is equally important to recognize when not to experiment, such as when changes are obviously beneficial, low-risk, time-sensitive, or lack adequate data. A well-structured experimentation lifecycle includes maintaining a backlog of ideas, designing experiments with clear metrics, and iterating on findings. Choosing the right tools involves options like standard experimentation, bringing your own analysis, or a hybrid approach. Effective decision-making relies on flexible statistical methods and consistent dashboards, while data slicing can enhance experiment value by identifying subgroup preferences. Building a strong experimentation practice requires discipline, streamlined processes, and a focus on high-uncertainty, high-impact changes.

Running experiments on high-traffic websites poses unique challenges due to the abundance of data, which can lead to unexpected pitfalls in traditional A/B testing methods. While reaching statistical significance quickly might seem advantageous, it can result in false positives and misinterpretation of minor effect sizes that have no real business impact. The infrastructure must support massive user loads without degrading performance, and issues like sample ratio mismatch can quickly invalidate results if not detected. To conduct successful high-traffic experiments, teams need to adopt tailored strategies, such as setting minimum detectable effect thresholds, optimizing infrastructure for performance, and implementing automated risk mitigation measures like circuit breakers and kill switches. LaunchDarkly offers solutions for high-traffic experimentation by integrating experimentation into development workflows, allowing for real-time monitoring, progressive rollouts, and instant rollbacks, ensuring that both engineering and product teams can derive actionable insights without compromising user experience or business outcomes.

A/B tests and Multi-Armed Bandits (MABs) are two distinct methodologies used in data-driven optimization, each serving different purposes. A/B tests involve a multi-step process where a hypothesis is formulated and validated through experimentation, making them ideal for precise measurement and long-term insights. They excel in providing accurate, transparent results and are immune to biases such as Simpson’s Paradox. In contrast, MABs are suitable for scenarios with ephemeral effects, where time is limited, such as short-term sales promotions or rapidly changing regulatory environments. They allow for quick optimization by reallocating traffic to leading variations, even though this may result in skewed estimates due to shifting traffic allocations. While A/B tests are considered the gold standard for accurate measurement, MABs offer a pragmatic solution when speed is critical, highlighting their value in specific contexts where rapid adaptation is required.

The development of AI applications, particularly those utilizing Large Language Models (LLMs), necessitates a shift from traditional programming paradigms. Unlike deterministic programming, LLMs are probabilistic and context-sensitive, producing varied outputs from similar inputs. This requires new development and quality assurance methods, such as creating hypotheses and testing the AI’s responses to ensure consistency. Developers must be familiar with advanced LLM concepts and techniques like Retrieval-Augmented Generation (RAG) and text-to-SQL for handling specific data types. Security, privacy, and bias protection are critical, with measures like data masking and prompt injection prevention. Evaluating AI applications involves creating test sets for consistent monitoring of accuracy, resource usage, and alignment with communication guidelines. Runtime experimentation with LLM configurations and prompts is vital for iterative development. Frameworks and tools like LaunchDarkly AI Configs facilitate this by allowing developers to manage AI configurations without redeploying applications, supporting the iterative nature of AI development.

LaunchDarkly has reimagined experimentation capabilities to better align with the workflow and mental model of product managers, aiming to address the complexities of the product management role. Traditionally, experimentation platforms have catered more to technical users, often sidelining product managers who are pivotal in driving experimentation success. LaunchDarkly's new system emphasizes a user experience that is clear, collaborative, and credible, integrating features like Event Explorer and Experiment Builder to enhance visibility, self-sufficiency, and ease of collaboration. The Event Explorer allows product managers to track and validate metric events, promoting independence from engineering tasks, while the Experiment Builder provides a visual, step-by-step interface for defining experiments and collaborating asynchronously with cross-functional teams. The Results View offers clear, research-backed visualizations and actionable insights, enabling product managers to confidently make decisions and communicate outcomes to stakeholders. Designed to fit seamlessly into existing workflows, this system reduces friction and empowers product managers to independently manage experiments, enhancing their ability to deliver real outcomes without relying on technical intermediaries.

LaunchDarkly's Guarded Rollouts for AI Configs offers a solution to the challenges of managing AI updates in production environments by providing automated safeguards and real-time feedback. The feature allows teams to update AI configurations—such as changing prompts, tweaking model parameters, or swapping providers—while gradually rolling out changes and setting guardrails based on key metrics like success rate and latency. If any metric crosses a predefined threshold, the system can automatically halt or revert the rollout, preventing user experience degradation. This approach contrasts with traditional, reactive risk management methods, which often involve manual interventions and are prone to stress and delays. By integrating with AI SDKs, LaunchDarkly tracks essential metrics without requiring additional instrumentation, allowing teams to confidently implement changes and ensure AI quality without constant monitoring.

Recovery Time Objective (RTO) and Recovery Point Objective (RPO) are critical metrics in disaster recovery, traditionally associated with major disruptions like hardware failures or data center issues. However, in modern software development, where code is deployed frequently, these metrics become crucial for every release due to the risks posed by bugs and software-induced incidents. RTO measures the allowable downtime before significant business impact, while RPO determines the permissible data loss. Traditional disaster recovery plans are often insufficient for rapid, continuous deployments, as they focus on large-scale events rather than frequent, software-related issues. The integration of feature flags and tools like LaunchDarkly in the development process can help teams achieve low RTO and RPO by allowing quick rollbacks and gradual rollouts, reducing the impact of software defects. This approach emphasizes prevention over cure, aiming to build resilience into the development workflow and minimize the need for costly, elaborate disaster recovery systems.

On June 12, a widespread outage affected Google Cloud Platform (GCP) users due to a bug in a new release impacting global load balancers, highlighting the critical role of feature flags in modern software deployment. Despite GCP's advanced engineering capabilities, the incident underscored the vulnerability inherent in rapid code deployment, as Continuous Integration/Continuous Deployment (CI/CD) pipelines can lead to loss of control once code is live. Feature flags provide a solution, offering a control plane that allows teams to manage changes incrementally, react to issues swiftly, and roll back deployments without redeployment. By using feature flags, GCP could have restricted the release to a small percentage of users, monitored metrics more effectively, and contained the issue without a full-scale outage. The narrative emphasizes that feature flags are not limited to front-end applications but are essential for managing infrastructure updates and high-risk production releases, providing teams with the confidence to deploy changes safely while maintaining the ability to swiftly recover from unforeseen problems.

The text discusses the challenges of dealing with "hallucinations" in Generative AI (GenAI) applications, emphasizing the importance of detecting and managing these errors to maintain user trust and product reliability. It highlights the risks of inaccurate AI outputs, which can lead to user frustration and even business loss. To address these issues, the text outlines three strategies: grounding responses using Retrieval-Augmented Generation (RAG) to provide accurate context, implementing guardrails to filter out risky outputs, and employing a secondary AI model to fact-check responses in real-time. These approaches aim to enhance the visibility and control over AI outputs in production environments, ensuring they are both accurate and safe. LaunchDarkly AI Configs is presented as a tool that aids in monitoring and optimizing AI systems by enabling teams to run tests, score responses, and automate responses to inaccuracies, all while maintaining delivery speed. The text concludes by inviting users to try out these configurations to build more trustworthy AI applications.

Engineering organizations often attempt to create their own feature flag systems, mistakenly believing them to be simple toggles, only to find themselves entangled in a complex and time-consuming maintenance burden. LaunchDarkly offers a comprehensive solution that alleviates these issues by providing a robust platform with critical features such as global support, structured experimentation, and guarded releases that allow for gradual rollouts and automated rollbacks. The platform simplifies the management of AI configurations by allowing changes without redeployment, ensuring compliance with standards like SOC 2 Type II and ISO certifications, and enabling scalability across various team sizes and environments. By integrating seamlessly with modern CI/CD systems and offering over 30 SDKs, LaunchDarkly allows teams to focus on delivering value rather than on the upkeep of internal tools, thereby enhancing productivity and enabling more efficient and reliable software development practices.

Large Language Model (LLM) inference, which involves generating output from trained models, is essential for producing the conversational and user-friendly interactions that have made LLMs increasingly popular in both consumer and enterprise contexts. As their usage grows, managing costs, reducing latency, and optimizing throughput have become crucial challenges. While cloud-based LLM services such as those from OpenAI do not require user-side optimization, organizations deploying their own models like LLAMA or Gemma need to experiment with various optimization techniques to enhance performance. This article, part of a broader series on AI application development, aims to delve into the specifics of optimizing LLM inference, offering practical guidance on experimentation and performance measurement.

Software teams often face challenges in ensuring the success of new features due to a lack of structured measurement, leading them to rely on intuition and anecdotal evidence, which are not always reliable. This can result in uncertainty, slow feedback loops, and decreased confidence in decision-making. The text emphasizes the importance of experimentation as a decision-making framework that provides clarity and insight by allowing teams to ask clear questions, define measurable outcomes, and evaluate impact rigorously. Experimentation helps in understanding user behavior, detecting unintended consequences, and iterating quickly, ultimately fostering trust across disciplines and leading to better product outcomes. However, the integration of experimentation into the development process requires aligning it with existing tools and data, designing experiments that reflect product goals, and ensuring data is trusted. LaunchDarkly offers a solution by embedding experimentation into feature flags and engineering workflows, enabling faster iteration and a more resilient product development process without the need for a dedicated lab.

In episode 729 of Software Engineering Daily, Jeff Meyerson talks with LaunchDarkly CEO and Co-founder Edith Harbaugh about feature flagging. Feature flags allow developers to deploy new software releases in a controlled fashion, enabling gradual rollouts and A/B testing. Edith shares insights on implementing feature flags, their benefits for product releases, and how they can be used for testing different aspects of software throughout an organization. LaunchDarkly's future involves expanding its offerings beyond developers to empower other teams within a company to manage features effectively.

In 2018, LaunchDarkly experienced significant growth, serving over 100 billion feature flags a day and increasing its team size by nearly 3X. The company received 2500 job applications in 2018 compared to less than 800 in 2017. To land a job at LaunchDarkly or any other company, the author suggests doing thorough research on the company and aligning your career goals with its values and offerings. The hiring process at LaunchDarkly includes a first call, a challenge exercise, an onsite interview, and a final decision based on both technical skills and alignment with company values.

LaunchDarkly has introduced a new feature that allows users to compare flag targeting rules across different environments within the same project. This helps maintain consistency and easily identify changes made for testing purposes. The functionality also includes confirmation safety checks to prevent accidental rule modifications. Accessible from the main dashboard, this feature aims to improve efficiency and control in managing flag targeting rules.