How to validate your Kubernetes Liveness Probes with Chaos Engineering

Liveness probes are crucial for maintaining the health of distributed applications running on Kubernetes by ensuring that malfunctioning services are automatically restarted. The blog post explores the use of Steadybit to verify the effectiveness of liveness probes through chaotic testing, specifically by injecting HTTP errors or delays into services to observe if Kubernetes restarts the affected pods. It discusses the implementation of a liveness probe using a Spring Boot application and the Health Actuator Endpoint, which triggers a pod restart if the health check fails. An experiment is conducted where a "Controller Delay" is introduced to test the probe's response, confirming that Kubernetes successfully restarts pods when the liveness check is unsuccessful. The post emphasizes the importance of chaotic testing in validating the functionality of liveness probes and suggests further experiments for comprehensive verification.