July 2026 Summaries

Dragonfly's new tagged chunks replication format significantly enhances the efficiency and correctness of the replication process by doubling throughput and reducing P99 latency by up to 87%. The previous system, which relied on a coarse serialization lock to ensure data consistency during replication, posed performance bottlenecks as it stalled other tasks while a network operation was pending. The introduction of tagged chunks allows for fine-grained locking, enabling large baseline entries to be split into non-contiguous chunks with envelopes for proper reconstruction. This new format ensures that changes can be streamed to a replica in real-time without compromising on data consistency, as it adheres to a "baseline before journal" rule that prevents correctness bugs. The tagged chunks format, only applied where necessary, optimizes memory use and parsing overhead by avoiding unnecessary tagging on small or non-data entries. Benchmark tests underscore the improvements, showing that the new format significantly boosts throughput and reduces latency, while allowing unrelated operations to progress even when others are stalled by network backpressure, leading to a marked increase in the number of completed write operations during full synchronization.

Dragonfly replaces Redis’s traditional dictionary implementation with DashTable, an extendible-hashing structure designed to reduce memory overhead and avoid whole-table rehashing. Redis grows by allocating a second, larger hash table and incrementally migrating entries, which spreads latency costs but creates temporary memory spikes and requires lookups across both tables during rehashing. DashTable instead organizes entries into fixed-size segments addressed through a directory; when insertion capacity is exhausted, it splits only the affected segment and redistributes at most 840 entries, while directory expansion moves pointers rather than key-value data. Its insertion process also attempts nearby buckets, entry displacement, stash buckets, and expired-key cleanup before splitting. The post estimates DashTable’s steady-state storage at about 18.6 bytes per entry, compared with Redis’s estimated 32–48 bytes including bucket overhead and rehash effects, and reports a benchmark in which Dragonfly used roughly 4.3 GB for 100 million keys versus Redis’s 7.5 GB, a reduction of about 43%. It also notes that Redis 7.x mitigates large rehash events through multiple internal dictionaries, whereas DashTable addresses the issue through localized segment-level growth.

Dragonfly has introduced a dictionary-based compression feature that effectively reduces memory usage for small records in queue-based frameworks like Celery and Sidekiq by 3-4 times. This new compression method leverages the common schema of small records, allowing them to be efficiently compressed as a group rather than individually. It is particularly beneficial for applications built on Python and Ruby frameworks, where asynchronous job queues are common. The feature is activated by setting a specific threshold, which determines when a list is large enough to train a compression dictionary. Once trained, this dictionary allows for consistent and efficient compression of list records, helping reduce memory consumption significantly. Benchmark tests on both Celery and Sidekiq revealed a substantial decrease in memory usage, with Celery's memory usage dropping to below 300 MiB and Sidekiq's to around 250 MiB when the feature was enabled. However, the feature's effectiveness depends on the uniformity of the list item schema and may not be as beneficial if the initial list used for dictionary training is not representative of future lists.

Instacart successfully enhanced its ad-serving feature store's performance by migrating from a managed Valkey solution to Dragonfly, resulting in a 50% reduction in latency and a 70% reduction in node usage within weeks. The migration was driven by the need to address the existing infrastructure's limitations, such as instability during cluster mutations, high tail latency due to data spread across numerous nodes, and escalating costs. Dragonfly's multi-threaded, shared-nothing architecture allowed Instacart to scale vertically on each node, leading to a more stable, cost-effective system. The transition involved setting up a Dragonfly cluster alongside the existing system, backfilling it with data, and gradually shifting traffic. This process required client-side tuning and engine-level improvements, ultimately achieving superior performance across all latency percentiles. The migration not only enhanced Instacart's operational control and efficiency but also provided significant infrastructure reductions, allowing for controlled upgrades and easier management. Tristan Fletcher, reflecting on the project, emphasized the importance of understanding data access patterns, abstracting storage layers, and being intentional about storage formats and key design to avoid compounded issues at scale.