December 2019 Summaries

LaunchDarkly has added two data centers in Asia-Pacific and Europe to improve initial connection times for its customers, resulting in nearly 50% decrease in client SDK initialization times in Europe and a whopping 90% in Asia-Pacific. The new data centers enable faster connections by routing traffic to the fastest data center based on an end user's location. This improvement is part of LaunchDarkly's larger initiative to enhance its enterprise-grade architecture, with plans to add more endpoints around the world and in the United States in 2020 for further performance improvements.

Developers working on modern platforms often need to consider concurrency issues when writing mobile apps or high-volume web services. The LaunchDarkly SDKs are designed for use in highly concurrent settings, but there are still a few edge cases worth keeping in mind. These scenarios involve avoiding mutability and performance bottlenecks from unnecessary repetitive flag evaluations. Developers should not modify user objects after passing them to the SDK, avoid modifying values within complex user attributes or flag values after receiving them, and be cautious of potential concurrency issues when using analytics events. By following these guidelines, developers can ensure their applications run smoothly and efficiently with LaunchDarkly SDKs.

The text provides a guide on configuring LaunchDarkly with Pivotal Cloud Foundry (PCF) applications. It outlines the prerequisites, initial setup, and steps to create a feature flag in LaunchDarkly. The guide also explains how to use the feature key created in application code to check which variation of a feature flag a user will receive. Additionally, it demonstrates how to validate your flag using the debugger in LaunchDarkly.

LaunchDarkly's REST API aims to maintain backward compatibility, but sometimes introduces breaking changes with new features and improvements. To accommodate this, they now support multiple API versions simultaneously, allowing users to migrate at their own pace without fear of unexpected behavior. Users can set the API version on a specific request by sending an LD-API-Version header or specifying it when creating an access token. Previously created tokens default to version 20160426. The company recommends explicitly setting the API version header in any client or integration built and only relying on the access token API version during manual testing.

Rosemary Wang, a Developer Advocate at HashiCorp, presented her experiences with testing infrastructure changes in production during the November Test in Production Meetup in Berlin, Germany. She discussed how feature flags, canary tests, and A/B tests can be used to control the blast radius when making network updates. Feature toggles or flags tell if something is "on" or "off," preserving the state of infrastructure. Canary testing involves releasing a new feature to a small group of users before rolling it out to everyone else, allowing for smoke testing and early identification of issues. A/B testing can be used in the infrastructure space by measuring how changes affect upstream service level objectives, such as processing speed or cost. These techniques help organize infrastructure blast radius and move from risk aversion to risk mitigation.

Effie Mouzeli, a Site Reliability Engineer (SRE) at the Wikimedia Foundation, discussed the steps taken to migrate Wikipedia from HVVM (HipHop Virtual Machine) to PHP7 while serving 100,000 requests per second and 17 billion page views per month. The migration involved cross-team collaboration, testing in production, and community support. Despite thorough pre-production testing, the team still encountered issues during live deployment, emphasizing the importance of testing in production to catch unforeseen user behavior.

In her talk at the Test in Production Meetup in Berlin, Heidi Waterhouse discussed how to build resilient software systems that can handle inevitable failures. She emphasized the importance of making releases smaller and faster, decoupling components so that failure in one part does not affect the whole system, and ensuring reversibility by avoiding deployments whenever possible. Observability is crucial for monitoring system performance, while breaking up tightly integrated systems into services helps manage dependencies. Partial successes or "successful failures" should be embraced as a reality of complex and connected systems.

Black Friday and Cyber Monday bring massive savings for shoppers, but also sleepless nights for engineering and operations teams as they battle incidents. Retailers experience varying degrees of outages during this period, with the most significant impact occurring between Thanksgiving Day and Cyber Monday. High traffic is one reason for these issues, but other factors include overburdened APIs, slow third-party technology integrations, sites heavy with graphics, or a failure to look at regional performance levels. Feature management can help retailers prepare for potential issues during peak shopping days by utilizing feature flags. This approach allows teams to quickly disable non-essential features and manage traffic spikes effectively.

In a recent talk at the Bay Area Test in Production Meetup, Ben Woskow emphasized that nothing in software development is perfect or invulnerable. He highlighted the inevitability of failure and encouraged teams to adopt a blameless culture, where they learn from mistakes rather than assign blame. This approach helps organizations reduce costly outages and recover from incidents faster. A blameless culture promotes teamwork, trust, and shared knowledge, which can lead to better incident response and recovery processes. Designing for failure involves monitoring, observability, alerting, response and recovery plans, considering customer impact, focusing on teams rather than individuals, and continuously improving processes and systems based on learnings from failures.