April 2018 Summaries

Remind, a communication platform used by over 70% of U.S. public schools, faced the challenge of handling peak user traffic during the back-to-school season, with over 27 million active users in 2017. To ensure seamless communication for teachers, students, and parents during this crucial period, Remind adopted Chaos Engineering tools, specifically utilizing Gremlin's Latency Gremlin, to simulate network delays and test system resilience. These chaos experiments helped Remind identify weaknesses and prepare additional technical resources, maintaining app stability even if Redis slowed down, and ensuring key features remained functional. The insights gained from these tests allowed for collaboration with the design team to enhance user experience, leading to Remind maintaining a top position in the App Store. Chaos Engineering has since become integral to Remind's engineering culture, with automated experiments incorporated into their deployment pipeline to preemptively address potential system failures.

APIs, often treated as secondary components within a company's infrastructure, are crucial for ensuring seamless user experiences, and their weaknesses can lead to significant performance issues or outages. Gremlin advocates for enhancing API resilience through chaos engineering, specifically by conducting API GameDays that simulate potential failures in a controlled environment to identify system vulnerabilities. Using the example of the "MyStatus" app, various chaos experiments, such as inducing memory attacks or network failures, are performed to test system reliability under stress. Additionally, the text emphasizes the importance of preparing for third-party API failures, as demonstrated by past outages affecting major services like Facebook, Instagram, and Twitter. To mitigate such risks, strategies like API virtualization and monitoring are recommended. Gremlin's platform offers tools for automated reliability testing, aiming to uncover hidden risks and improve system resilience before they impact users, supported by features like custom reliability test suites and ongoing progress measurement.