Our Kubernetes Operator Didn’t Scale, So We Rebuilt It

The blog post discusses the challenges and solutions associated with scaling a Kubernetes operator for secrets management, focusing on the transition from an outdated architecture to a more efficient reference-based design. Initially, the Kubernetes operator struggled with scalability due to its monolithic architecture, which required each resource to independently handle authentication and connection, leading to high memory consumption and inefficiencies. To address these issues, the operator was rearchitected to separate connection, authentication, and synchronization into distinct resources, mirroring the External Secrets Operator's pattern. This new architecture reduces memory usage, simplifies configuration changes, and improves developer experience by allowing shared authentication and connection resources, thus solving previous scalability problems. The post also highlights additional enhancements in the new version, such as support for multiple source paths and targets, and readiness status reporting for resources, ensuring seamless management of Kubernetes secrets across varied infrastructures.