Moving persistent data out of Redis

GitHub historically used Redis in two primary ways: as a transient LRU cache for storing computation results and as a persistent store for data not stored elsewhere. Recently, the company decided to disable Redis persistence to reduce operational costs, leverage MySQL expertise, and improve performance by eliminating I/O latency. The transition involved replacing persistent Redis with GitHub::KV, a MySQL key/value store, for most use cases. Migrating activity feeds posed challenges due to high write operations, necessitating changes in how data was written and read. By batching and throttling writes and experimenting with schema designs, GitHub successfully migrated feeds to MySQL without adding new servers. The data-driven approach led to a reduction in write operations and ensured the system's capacity to handle the load. The project highlighted the need for a solution that works across GitHub's platforms, resulting in improved efficiency and performance. The Platform and Infrastructure teams played a crucial role in this transition, and GitHub is seeking engineers to join their efforts in building a resilient platform.