September 2021 Summaries

Memory attacks, which consume a specified amount of system memory, are essential for understanding and optimizing system performance, stability, and cost management. These attacks allow engineers to simulate memory-intensive conditions, aiding in infrastructure right-sizing, preparation for out-of-memory scenarios, and testing system resilience during high-traffic events. By setting the magnitude of a memory attack either in megabytes, gigabytes, or as a percentage of total memory, teams can observe the effects on system stability and make necessary adjustments. The process involves using tools like Gremlin to configure and monitor attacks, ensuring that memory usage is within safe limits to prevent system crashes. Conducting these experiments helps organizations validate system robustness, reduce operating expenses, and streamline the transition to cloud environments by mimicking real-world conditions.

In this podcast episode, John Martinez, Director of Cloud R&D at Palo Alto Networks, discusses with host Jason the intricacies of FinOps and its impact on cloud cost optimization, particularly in a multi-cloud environment. John shares his insights on employing a "crawl, walk, run" approach to FinOps and emphasizes the importance of understanding where and how cloud spending occurs for effective financial management. They explore the challenges of security and reliability in a multi-cloud setup, drawing parallels between security practices and chaos engineering principles. John also highlights his commitment to diversity and inclusion through his involvement with Latinx in Tech and Techqueria, illustrating his dedication to fostering diverse communities in tech. The conversation touches on the importance of monitoring and alerting on misconfigurations to preempt potential security risks, advocating for embedding security measures early in the development process.

CPU attacks, commonly employed by Gremlin users, are strategic exercises to consume CPU capacity on various platforms like hosts, containers, and Kubernetes resources, revealing potential system vulnerabilities such as poor performance and instability. These attacks are crucial for ensuring systems maintain reliability even under heavy CPU usage, which can be unpredictable due to factors like increased production demands or unexpected traffic spikes. By simulating scenarios such as stress testing and noisy neighbor environments, CPU attacks help validate system stability for high-traffic events and optimize infrastructure capacity, thus reducing operating costs. Gremlin offers fine-grained control over these attacks, allowing users to specify CPU consumption percentage and the number of cores involved, thereby enabling tailored tests that can scale from small, focused experiments to larger, comprehensive ones. Monitoring CPU usage during these attacks is essential to understanding system performance, and Gremlin provides tools for visualizing these metrics. The process encourages gradual scaling of the attack's scope and magnitude to safely assess and improve system resilience, with the results documented for organizational learning and improvement.

In a podcast episode titled "Break Things on Purpose," Omar Marrero, Chaos and Performance Engineering Lead at Kessel Run, discusses chaos engineering within the Department of Defense (DOD). Kessel Run, described as a software factory within the DOD, aims to rapidly deliver combat capabilities and award-winning software to warfighters by integrating DevSecOps practices. Marrero shares insights on adopting chaos engineering to enhance system reliability by intentionally introducing disruptions to test the resilience of military software systems. He emphasizes starting with simple tests and scaling up while gaining leadership support by demonstrating the value of chaos engineering. Kessel Run has developed a team and tool called Bowcaster, which employs Chaos Plus—an approach combining chaos, performance, and cybersecurity testing—to ensure robust and fault-tolerant systems for the DOD. The discussion highlights the importance of iterative testing and collaboration with teams to address potential failure points, ultimately contributing to the DOD's broader DevSecOps transformation.