Managing long-running tasks with Celery and Kubernetes (or, keeping your sanity during deploys)

Merge, a company offering a unified API for integrating various platforms, addresses the challenge of managing long-running tasks alongside code deployments on Kubernetes clusters. The issue arises when tasks, often dependent on third-party API rate limits, run concurrently with necessary code updates, posing a risk to operational efficiency. To tackle this, Merge transitioned from a Recreate to a RollingUpdate deployment strategy in Kubernetes, allowing for the management of pod availability through configurations like maxSurge and maxUnavailable. This approach ensures that long-running tasks can complete without disruption while enabling seamless deployment of new code. The solution contrasts with a similar challenge faced by Brex, which relied on different configurations due to varying software behaviors. Merge also integrates Helm for managing Kubernetes configurations, underscoring the complexity of achieving harmony between infrastructure and software processes.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.