In March 2023, Datadog experienced a significant outage due to an unsupervised global update, revealing critical limitations in their systems' ability to handle failures gracefully. The incident underscored the need for a shift from preventing failures entirely to embracing strategies for graceful degradation, ensuring partial functionality even in the face of significant disruptions. Datadog's response involved reevaluating their system designs, focusing on data persistence, prioritizing real-time data processing, and implementing chaos testing to validate improvements. They recognized the importance of avoiding global control systems and reducing technical debt to prevent complex failure modes. By prioritizing the end-user experience and building systems that can adapt and recover quickly, Datadog has reduced the impact and duration of incidents, improving their overall resilience and reliability. This shift has resulted in a noticeable decrease in significant incidents and faster recovery times for their products, reflecting a more robust and customer-focused infrastructure.