October 2018 Summaries

In September, Brandon Leach, Senior Engineering Manager at Lookout, discussed increasing deployment safety with metric-based canaries during a Test in Production Meetup. Lookout is focused on mobile end-point security and has over 100 million devices registered. The company recently rebuilt its service delivery tooling for continuous delivery using Spinnaker, a service delivery platform created by Netflix. This transition resulted in significant improvements in deployment efficiency and service stability. Lookout's next goal was to achieve continuous deployment through testing in production. To do this, they began looking for ways to automate metric-based deployments in Spinnaker. Through collaboration with Armory, Lookout developed Barometer Service, which integrates with Spinnaker and uses time series data from monitoring platforms like DataDog to orchestrate metric-based canaries. The Barometer service compares the metrics of a baseline deployment with those of a canary deployment. If the metrics fall outside a defined deviation, the canary fails. Once the canary stage is completed, both the canary and the baseline server groups are destroyed, and the pipeline continues either as a success or a failure. Lookout has learned that it takes time to identify the appropriate metrics for monitoring deployments, and too many metrics can cause false positives. While metric-based canaries may initially slow down deployment times, they ultimately save time by automating the evaluation process. The company is currently in the process of migrating its canary deployments from Barometer to Kayenta, a platform developed through collaboration between Netflix and Google for automated canary analysis in Spinnaker.

The text discusses the importance of diversity in technology and shares personal experiences from attending the Grace Hopper Celebration (GHC) conference and working at LaunchDarkly as a woman in tech. Key points include intellectual honesty, empowering voices, blamelessness, learning sessions, and diverse backgrounds. The author emphasizes that what really matters is the ability to learn and make the most out of one's knowledge, rather than presuming incompetence based on different background experiences.

In this talk, Tim Wong discusses the use of feature flagging for managing operational risks in DevOps. He explains how feature flags can be used for more than just enabling or disabling features and highlights their potential to mitigate risk around complex changes. Wong provides examples of using feature flags for database migration, safety valves, and dynamic configurations. He emphasizes the importance of understanding how these tools work and monitoring their impact on system performance. Overall, this talk offers insights into innovative ways to use feature flagging in DevOps practices.

LaunchDarkly, a leading Feature Management company, has announced that over 500 businesses across the globe are now using its platform to build software applications and release new features faster with less risk. The platform allows businesses to decouple the deployment of software features from their availability to users, expediting the development process and gaining fine-grained control over the customer experience. LaunchDarkly's customers include Microsoft, Pitney Bowes, Intuit, Atlassian, Invision, Procore, and PluralSight. The company was founded in 2014 by Edith Harbaugh (CEO) and John Kodumal (CTO), with headquarters in Oakland, CA.

LaunchDarkly is a tool that assists various parts of a product delivery organization, including engineers, DevOps, PMs, and customer success teams, to perform their tasks faster with less risk through feature flags. The platform has recently introduced a bulk invite feature, making it easier for users to share the tool with multiple team members at once by assigning them appropriate roles in one quick step. This new feature promotes flagging as a best practice within organizations and ensures organizational security by enforcing multi-factor authentication or enabling SSO. The update allows teams to experience benefits such as separating deploys from releases, testing in production, and progressive delivery more efficiently.

In this talk, TR Jordan from Turbine Labs discusses how his team tests configurations in production using Slack's Turbine Labs service mesh based on Envoy. He emphasizes the importance of treating configuration as code and highlights three key points: configs tend to be deeply repetitive, they vary by environment, and not all changes are driven by humans. Jordan suggests techniques for testing complex configurations in production, such as phasing rollouts and separating human-driven intentional config from application state. He concludes that simplifying the configuration process can lead to a more manageable system with effective confidence building through testing in production.

LaunchDarkly holds a semi-regular hackathon called "Moonshots" where employees from all disciplines can step outside their daily work, try something new, and potentially affect the company's future direction. The Moonshots event encourages collaboration or solo efforts, requires preparation of a project manifesto beforehand, and demands presentations afterward to share outcomes. This edition of Moonshots resulted in several innovative projects, including cloning flags functionality, LaunchDarkly SDK integration with Arduino, an SDK test dashboard, a children's guide to feature flags, and a tool for comparing flags across environments. These projects showcase the company's commitment to continuous improvement and innovation.

In episode 49 of To Be Continuous, Edith and Paul delve into the topic of leading and managing a developer product to achieve success. The podcast is brought to you by Heavybit, an organization that supports developer-focused companies. Additionally, they offer a library filled with educational talks from industry leaders and fellow company founders. You can listen to this episode on SoundCloud at the provided link.

At GitHub Universe, GitHub introduced a new feature called GitHub Actions that allows users to implement custom logic without creating an app. With this tool, workflows are defined in repositories and triggered by events on GitHub. GitHub Actions can be used for various tasks such as publishing npm modules, sending SMS alerts when urgent issues are created, or deploying production-ready code. The integration of LaunchDarkly with GitHub Actions allows developers to automate workflows while maintaining control through the use of feature flags. This combination enables teams to manage hundreds or even thousands of flags effectively and efficiently.

The text discusses an update to a Visual Studio Code extension that aims to improve the efficiency of developers by reducing context-switching distractions. Key features include flag key autocomplete, displaying flag details on cursor hover, and an "Open in LaunchDarkly" command for quick access to feature flags in a browser tab. To use this extension, users can search for "LaunchDarkly" within Visual Studio Code or download it from the marketplace, followed by configuring vscode with the required settings as per the extension's README.

LaunchDarkly introduces a new feature called Clone Flags that simplifies the process of creating multiple similar feature flags for large projects. This functionality allows users to copy the full targeting configuration from an original flag to a new one, making it easier to create flag templates or manage flags related to a single project. The cloned flags are differentiated in audit logs and can also be created programmatically through the API. Clone Flags is available under the Settings tab for all LaunchDarkly flags.

Feature flagging systems can either fail safe or fail broken when they go down. In a fail-safe system, no updates are made and everything continues to operate as usual, with users seeing no changes after the system comes back up. This is possible because flag states are set and maintained on the client side. In a fail-broken system, user experience may be affected by failed calls, timeouts, or error messages. While there are reasons to use a fail-broken system, most applications benefit from failing safe. Testing failure states is crucial for ensuring an application's expected behavior during unknown-unknown failure scenarios.

LaunchDarkly recently hosted a meetup focused on chaos engineering, featuring engineers from Netflix, LinkedIn, and Gremlin. The event aimed to discuss how teams approach testing in production and share best practices for doing so safely. Honeycomb CEO Charity Majors emphasized the importance of resilience testing and acknowledging unknown unknowns when building systems. Netflix's Nora Jones discussed a chaos engineering tool developed by her team, while LinkedIn's Ted Strzalkowski shared how his team focuses on providing a comprehensive, automated, and measurable resiliency feedback loop. Gremlin's Pat Higgins talked about holistic thinking around failure and the development of user experience values in their platform. The panel discussed various aspects of chaos engineering, including the importance of order, monitoring, and safety measures. They also touched upon the challenges of bridging chasms between different stages of implementing chaos engineering within a company and obtaining buy-in for the long process. The engineers shared insights on how they prevent thundering herds of chaos experiments from taking over production systems and ensuring that failure mitigation strategies are practiced regularly to create a culture around it.

Feature flagging is a technique used in software development to control the release of new features by using simple on-off switches or more complex multivariate flags. Binary flags are commonly used for continuous delivery, progressive delivery, and testing in production. However, more expressive flags like numeric and JSON flags can be useful for controlling complex scenarios beyond just toggling a feature on or off. Numeric flags can be used as configuration values or to test multiple designs of a feature. JSON flags allow developers to decouple the precise behavior of a feature from its development and deployment, enabling real-time adjustments based on data and customer feedback.

LaunchDarkly has added native Redis feature store support to its .NET SDK, providing a persistent store for feature flag configurations. The Redis feature store can be used in two ways: with a regular connection to LaunchDarkly or with the LaunchDarkly Relay Proxy in daemon mode. In both scenarios, the feature flags are stored in Redis and retrieved using an in-memory read-through cache for improved performance. The Redis feature store is available as a separate NuGet package and its source code is published online. This addition complements the existing Redis support across all other server-side SDKs offered by LaunchDarkly.