Building a simple dynamic job scheduler with Sidekiq

Tines experienced an issue with their job scheduler, which impacted the execution of scheduled Actions for multiple customers, prompting a thorough investigation and subsequent system upgrade. The problem arose from a race condition during a rolling deployment, where a conflict between processes led to a gap in the scheduling of jobs. This was attributed to the use of the sidekiq-scheduler and its interaction with Redis, causing scheduled actions to be missed. In response, Tines developed a new scheduler by leveraging existing tools and patterns, focusing on simplicity, speed, and soundness. The new system relies on a database as the primary source of truth for schedules and employs a "fan out" job to ensure all scheduled Actions are processed reliably. This change has enhanced the reliability of dynamically scheduled jobs, eliminated missed runs, and has been successfully rolled out to all customers without further issues.