June 2025 Summaries

AI has become a significant investment for companies, and maintaining the reliability of AI applications requires both traditional and innovative approaches. Despite AI applications running on existing infrastructure, they introduce complexities such as new traffic patterns and dependencies, necessitating adjustments in operational strategies. Key challenges include ensuring both the availability of AI systems and the accuracy of their responses, which requires collaboration between DevOps and AI engineers. As AI continues to evolve, organizations must balance enabling new technologies while setting appropriate guardrails and testing processes to minimize customer impact. Engineering teams play a crucial role in maintaining AI reliability by defining metrics, conducting resilience testing, and integrating AI specialists into incident response plans. The ongoing development of best practices, such as GPU testing and specific SLOs, underscores the need for continuous learning and adaptation in the field of AI operations.

The recent Google Cloud Platform (GCP) outage on June 12th, 2025, which lasted over two hours and impacted more than 80 Google services, underscored the vast interdependencies inherent in modern architectures and the potential for widespread disruption. Companies unaffected directly by GCP still encountered issues due to their reliance on vendors that use GCP, as exemplified by Cloudflare's service interruption. To mitigate such risks, organizations are encouraged to adopt rigorous testing practices, including resilience testing, "Blackhole" experiments, and comprehensive backup tests to ensure their systems can withstand various failure scenarios. Regular testing of third-party dependencies and backups is crucial, as it allows companies to identify and rectify vulnerabilities before they lead to critical failures. This proactive approach not only prepares systems for inevitable outages but also minimizes customer impact, ultimately enhancing system reliability and resilience.