May 2019 Summaries

Gremlin has introduced upgrades to its infrastructure attack options to enhance configurability, reliability, and ease of use, focusing on CPU, disk, and memory attacks. These updates allow for more realistic stress testing by enabling users to impact all CPU cores at once, specify CPU utilization percentages, and consume a percentage of available memory rather than a fixed amount. The improvements aim to help users verify their monitoring and alerting systems, test autoscaling functions, and ensure application stability under constraints such as full storage or high memory usage. The updates encourage users to challenge their assumptions about service reliability by simulating real-world stress scenarios, ultimately aiming to reveal and fix potential availability risks before they affect end users. Gremlin's platform facilitates the discovery of hidden risks in systems, offering a free 30-day trial to explore these features.

The article provides insights and strategies for preparing systems to handle peak traffic events, which can significantly impact businesses in various industries, such as retail, financial services, software, news organizations, gaming, and education. It emphasizes the importance of having experienced Incident Managers on Call (IMOCs) during high-traffic periods, creating a comprehensive one-page document with critical information, learning from past postmortems, and testing in production to identify potential failures. Understanding and addressing problem services early and preparing well in advance are also highlighted as crucial steps to ensure system reliability during such events. The author, drawing from a decade of experience, shares that starting preparations early increases confidence in system readiness and encourages sharing tips within a community of engineers for continuous improvement.

In episode two of the "Break Things on Purpose" podcast, hosts Rich Burroughs and Jacob Plicque interview Michael Kehoe, a Staff Site Reliability Engineer at LinkedIn, about the company's approach to Chaos Engineering. Kehoe discusses LinkedIn's Chaos Engineering project, Waterbear, which includes tools like Fire Drill, LinkedOut, and D2 Tuner to test and enhance system reliability without impacting user experience. He highlights the importance of understanding system behavior under stress to improve resilience and mentions his background at NASA, which influenced his focus on reliability. The conversation emphasizes the significance of communication between teams to effectively implement Chaos Engineering and the potential for such practices to break down silos and foster a culture of operational excellence. Kehoe also touches on the future of Chaos Engineering, suggesting it will increasingly be integrated into API layers and involve human factors like personnel availability.