Attack of the mutant tags! Or why tag mutability is a real security threat

Tag mutability in container environments, while convenient, poses significant security risks by allowing image tags to change unexpectedly, which can lead to deployment issues and security vulnerabilities such as the Time-of-check vs. Time-of-use (TOCTOU) problem. Tags in container registries act as pointers to image manifests, and mutable tags can lead to non-deterministic deployments, where the image expected is not the one deployed, potentially allowing malicious images to bypass security checks. Despite the benefits of mutable tags, such as simplifying version tracking and deployment processes, they can cause unpredictable behavior and security threats if not managed properly. Using immutable tags could mitigate these risks, but support for them is limited in many registries. To ensure secure and repeatable deployments, it's recommended to use image digests instead of tags, employ strict security practices, and leverage tools like Sysdig Secure's image scanner and admission controller to validate and monitor Kubernetes deployments, ensuring the exact image version is used consistently across environments.