November 2023 Summaries

Failure Flags is a software-level library developed by Gremlin to enhance the testing of software failure modes without compromising security or requiring extensive infrastructure changes. Unlike traditional Chaos Engineering, which focuses on infrastructure, Failure Flags allows for simulating fault conditions within applications, thereby improving the reliability of deployments. By using the Failure Flags SDK, developers can integrate fault-injection points into their code, which can then be used to run experiments that simulate various failure scenarios such as dependency loss, latency, and API gateway issues. This approach is designed to be safe and portable, ensuring that test configurations can be left in place without impacting application performance when not conducting experiments. Failure Flags can be used across multiple platforms and integrates with existing CI/CD and observability tools, making it a versatile addition to any development workflow.

Gremlin has introduced Custom Reliability Test Suites, Scoring, and Dashboards to enhance its Reliability Management platform, allowing organizations to customize their reliability standards and measure progress more effectively. These new features enable companies to define tests that align with their specific compliance needs, such as OCC or DORA, and industry frameworks, while also addressing unique system vulnerabilities through tailored fault injection scenarios. The platform now offers a custom reliability score that integrates these tests, providing a comprehensive view of an organization's reliability posture and allowing comparisons across services and teams. Executive dashboards further enhance visibility by consolidating reliability data into a single interface, helping to identify and mitigate risks organization-wide. This proactive approach shifts the focus from reactive incident management to preemptive reliability assurance, reducing potential outage costs and improving overall infrastructure resilience.

Organizations should treat reliability risks with the same proactive approach as security vulnerabilities, employing regular scanning and testing to prevent costly outages before they occur. Despite recognizing the importance of addressing security vulnerabilities to avoid exploitation, many companies overlook reliability risks until they result in incidents or outages, which can be expensive. To mitigate such risks, organizations can use tools like Gremlin to detect common reliability concerns, conduct tests for specific failure modes, and employ Chaos Engineering experiments to identify unique system vulnerabilities. These strategies involve collaboration with service owners to address detected issues and continually test systems to find new risks, ultimately ensuring systems maintain high availability and reliability.