March 2022 Summaries

DNS attacks, often humorously blamed for outages, are critical for assessing and improving the resilience of online services, as demonstrated by Gremlin's DNS attack tool. The Domain Name System (DNS) is essential for translating human-friendly domain names into IP addresses, and disruptions can lead to widespread service failures. Gremlin's DNS attack simulates DNS outages by blocking DNS traffic, allowing organizations to test their systems' ability to withstand and recover from such incidents. The tool offers customizable parameters for controlling the attack's magnitude and blast radius, including targeting specific IP addresses, network protocols, and service providers. Running these attacks helps verify the effectiveness of mitigation strategies like fallback DNS servers and multiple DNS providers, ensuring systems remain operational during real-world DNS failures. By experimenting with DNS attacks, teams can identify vulnerabilities, track improvements, and demonstrate the value of resilience strategies to stakeholders.

In this podcast episode, Chris Martello, the manager of application performance at Cengage, discusses his journey from being a middle school science teacher to a chaos engineering enthusiast, emphasizing the importance of chaos engineering in enhancing software reliability. He explains how Cengage leverages chaos engineering to prepare for peak traffic events typical in the higher education sector, which aligns with academic calendars. Martello highlights the significance of practicing chaos through collaborative fire drills, which improve communication and reduce mean time to resolution during actual outages. He recounts how the "Day of Darkness," an 18-hour service outage, led to the implementation of regular chaos testing, involving 16 different teams to ensure system stability and performance. Martello also underlines the critical role of customer support in responding to performance issues, illustrating how chaos testing has become an integral part of Cengage's engineering culture, ultimately contributing to a robust and reliable user experience.

Packet Loss attacks, as facilitated by Gremlin, simulate network disruptions by deliberately dropping or corrupting a percentage of outbound packets to test the resilience of applications and systems under adverse conditions. These attacks help developers understand how their systems perform in real-world scenarios where networks can be unreliable or oversaturated, a key consideration in an increasingly remote and streaming-reliant world. By adjusting parameters such as the percentage of packet loss, specific IP addresses, ports, and protocols, users can tailor the attacks to match potential network issues faced by their systems, allowing for targeted and scalable testing. The process involves monitoring system performance during these attacks to gather data and insights, which can then be used to improve network robustness and ensure a seamless user experience even in challenging conditions. Gremlin's platform supports running Packet Loss attacks at varying scales, from single hosts to multiple systems, providing a comprehensive tool for anticipating and mitigating potential network-related issues before they impact end users.

Scaling applications and environments effectively requires a dual approach that combines performance engineering with resilience, underscoring the importance of Chaos Engineering. While performance engineering ensures systems can handle increased user loads, it often does so under ideal conditions that don't account for real-world challenges, such as infrastructure outages. The analogy of the Burj Khalifa illustrates that a system's ability to scale is incomplete without ensuring reliability and resilience. Gremlin advocates for integrating Chaos Engineering to address these challenges proactively, suggesting that this combination empowers enterprises to reduce incidents, increase availability, and maintain robust systems that delight end-users. By merging these approaches, companies can not only scale efficiently but also ensure that their systems remain operational under adverse conditions.

In a crossover podcast episode, Julie Gunderson and Mandi Walls engage with Alex Solomon, co-founder of PagerDuty, and Kolton Andrus, co-founder of Gremlin, to discuss the founding stories and cultural philosophies of their companies. They delve into their experiences transitioning from large corporations like Amazon and Netflix to startups, highlighting the importance of building resilient systems and the challenges of changing organizational culture. The conversation touches on the evolution of DevOps practices, the necessity of modernizing incident management, and the adoption of Chaos Engineering to enhance system reliability. They address common misconceptions and provide advice for organizations hesitant to adopt practices utilized by tech giants like Amazon and Netflix. The episode emphasizes the gradual journey towards modernization, encouraging companies to take small, impactful steps to improve their operational maturity.

As cloud-native systems become more prevalent, user tolerance for latency decreases, prompting the need for testing application performance under adverse network conditions. A Latency attack injects delays into outbound network traffic, helping developers build responsive and fault-tolerant applications by emulating packet latency using Quality of Service (QoS) facilities in Linux or a custom driver in Windows. This method does not alter firewall rules and allows for targeted testing by configuring parameters such as IP addresses, hostnames, ports, and protocols. By conducting Latency attacks, developers can ensure that applications perform reliably under various network conditions, validate retry and timeout thresholds, and evaluate load balancer configurations. Gremlin's platform facilitates these tests, allowing users to simulate network degradation across multiple hosts and services, thereby maximizing performance and user experience. Additionally, scenarios and health checks can be employed to monitor system availability during these tests, and Gremlin offers pre-built scenarios designed by reliability experts to enhance testing rigor and effectiveness.