March 2025 Summaries

Failure Flags, a tool designed to enhance application-level reliability through resilience testing, has transitioned out of its Beta phase and is now generally available. Over the past year and a half, it has been utilized by customers to conduct numerous tests on applications running on platforms such as AWS Lambda, Kubernetes, and Istio Service Mesh, using components like the Gremlin SaaS API and SDKs integrated into application code. The tool allows users to simulate application errors, latency, and data issues, helping teams manage common challenges like incorrect data and API changes. With SDKs available for popular languages such as Node.js, Python, Java, Go, and now .NET, Failure Flags enables creating experiments through a user-friendly Gremlin UI. This updated interface allows for easy configuration of tests, helping to standardize and scale resilience efforts across organizations. As part of Gremlin's automated reliability platform, Failure Flags aims to identify and mitigate availability risks before they affect users, with further improvements and optimizations already underway.

A major apparel company faced the challenge of testing the failover capability of their AWS Lambda-based payment application between regions without access to underlying infrastructure, a common issue with serverless models. They employed Failure Flags, a tool designed for application-level testing in managed environments, which allowed them to simulate a regional outage and test the failover process within 30 minutes. This test not only confirmed the system's resilience, potentially saving millions in sales, but also revealed areas for performance improvement. Encouraged by these results, the company plans to expand the use of Gremlin's reliability tests across all deployment environments, including Lambda, EKS, and ECS, to further enhance system resilience and performance.

AI-as-a-service (AIaaS) providers such as OpenAI, AWS, and Google Cloud have revolutionized the accessibility of AI models, enabling a wide range of applications to incorporate AI functionalities with ease. However, these services are not immune to failures, which can disrupt dependent applications. To ensure resilience, developers can implement strategies like circuit breakers, alternative routing, and fallback mechanisms. Gremlin, a tool for testing service reliability, allows users to simulate AIaaS failures to observe and strengthen their systems' responses. It offers tests for network outages, latency, and certificate expirations, enabling developers to identify and address potential vulnerabilities before they affect users. Regular testing and monitoring through tools like Gremlin can enhance the reliability of AI-powered services, ensuring consistent performance even during outages.