Building a Background Processing Pipeline with Dragonfly

This blog post demonstrates how to use Redis Lists to build a background processing pipeline with Dragonfly, an in-memory data store built for modern application workloads. The use case is a marketing automation system that ingests user events and sends notifications based on some of them - for example, a welcome email when a user registers on a website. The solution includes two applications: a producer application generating fake user registrations and adding them to a task queue, and a worker application picking tasks off the queue and sending emails corresponding to them. Both applications are written in Go and make use of the gocelery library. The pipeline can be scaled horizontally by simply adding more worker applications.