November 2018 Summaries

Testing code under various flags is crucial in ensuring software quality and functionality. There are five main approaches to achieve this: creating a dedicated LaunchDarkly testing environment, setting the SDK to offline mode, using a Redis cache with LDD mode enabled, engineering a mock SDK for testing, or building a custom feature store for testing. Each approach has its own advantages and disadvantages, depending on factors such as scalability, control over flag configurations, and network requests made during testing.

Many software teams in the U.S. are implementing "deploy freezes" during Thanksgiving week to reduce the risk of issues occurring at inconvenient times. However, some companies like LaunchDarkly continue deploying by using feature flags to mitigate risks and maintain peace of mind. This approach allows them to avoid accumulating un-deployed code and ensures they remain critical for teams that can't take time off during the holiday season. Last week, LaunchDarkly conducted an internal "focus week" where engineers tackled various tasks such as cross-service tracing, upgrading ElasticSearch clusters, and improving user experience. This work will be deployed during Thanksgiving week, allowing the team to focus on other aspects of their jobs without worrying about code deployment issues.

In October, a special Test in Production session was held with Honeycomb.io where speakers shared their scariest stories of operational outages and other horrors. Eric Pollman, CTO and co-founder of ClearBrain, discussed two such experiences from his time at Google. The first incident involved Google Ads going fully offline for 90+ minutes due to a bad data push that caused servers to crash. The second story was about a "zombie haunted" pipeline that kept developers awake late into the night while processing large amounts of data. These experiences taught valuable lessons, such as checking multiple metrics and automating processes to prevent similar issues in the future.

In a special Test in Production session with Honeycomb.io, various speakers shared their horror stories of operational outages and other scary events. Marc Devens, Director of Product at BrandVerge, discussed two such incidents. The first involved Appeagle, a company that provides price management software for third-party sellers on Amazon, eBay, and Walmart. A major pricing error occurred when items were listed as one cent instead of thousands of dollars. The second incident involved the launch of multiple features with an offshore contract team after months of only shipping fixes. Devens emphasized the importance of testing in production and ensuring that disasters never happen again by implementing tests, safety checks, and gating mechanisms to control rollouts and enable quick reversals if necessary.

In October, a special Test in Production session was cohosted by Honeycomb.io and featured speakers sharing their scariest stories of operational outages and other horrors. One such story was told by Paul Biggar, CTO and Co-founder of Dark, who recounted how a massive security breach nearly ended his previous company, CircleCI. The incident began with the discovery of a missing AWS identity and access management key, which led to a full-scale security investigation. CircleCI was affected by the Mongo HQ security breach, as they were a customer of the compromised database provider. After recycling all their keys, CircleCI faced the challenge of communicating with customers and restoring service while ensuring safety. Jesse Robbins, CEO of Orion and former Master of Disaster at Amazon.com, provided valuable guidance on how to handle the situation effectively. Throughout the incident, transparency was prioritized, with regular updates being shared with customers via email and social media. The company managed to retain almost all its customers despite the scare, thanks in part to their transparent response. This experience taught Paul Biggar valuable lessons about the importance of encryption at rest and maintaining transparency during crisis management.

Atlassian, a leading software company known for its collaboration platforms such as Jira Software and Trello, has adopted feature management practices to improve team performance and customer satisfaction. The company uses LaunchDarkly, a feature flagging and management platform, to separate code deployments from feature releases, allowing teams to ship code at any time with features hidden until they're ready. This approach enables more efficient collaboration between engineering, design, and product management teams, as well as continuous delivery of value to customers. The integration of LaunchDarkly with Jira Software has further streamlined the development process by providing operations teams with better control over feature flags and reducing context switching.