November 2025 Summaries

Dragonfly's SSD Data Tiering feature innovatively expands in-memory databases by integrating SSD storage to manage massive datasets cost-effectively, maintaining high performance and low latency. This hybrid approach addresses the limitations of RAM's high costs by utilizing SSDs, which have become faster and more affordable, to store less frequently accessed data on disk while keeping frequently accessed data in memory. Dragonfly uses a shared-nothing architecture where each thread manages its own storage file, storing only entry data on disk while metadata remains in RAM, ensuring efficient data retrieval. It employs io_uring for asynchronous I/O operations, bypassing the Linux page cache to optimize memory usage and prevent redundant data copying. The system intelligently identifies and manages "hot," "cold," and "cooled" entries, with a background process offloading rarely accessed data to disk and promoting it back to RAM when needed, thus balancing speed with capacity. Benchmarks show that Dragonfly outperforms ElastiCache in read throughput and maintains lower latency under load, highlighting its potential for large-scale data management with reduced infrastructure costs.

Dragonfly has introduced a native support system for geospatial indexing and querying, leveraging the R-Tree data structure to efficiently handle geographical data. R-Trees are designed for multi-dimensional data, allowing for effective range and nearest-neighbor searches by organizing data hierarchically and minimizing unnecessary search space. Dragonfly's implementation involves using the Boost Geometry library with a linear split strategy for its R-Tree configuration, which allows for simple insertion and deletion operations. However, to enable radius-based searches, Dragonfly developed a custom approach involving the construction of a temporary polygon and a minimum bounding box, followed by Haversine distance calculations. Performance benchmarks show Dragonfly outperforming Redis in terms of request processing speed and scalability, making it a strong candidate for high-volume geospatial search applications. As data volumes grow, Dragonfly's combination of R-Tree indexing and multi-threaded processing offers significant advantages, positioning it as a more efficient and scalable solution compared to RediSearch.

The in-memory data infrastructure landscape has undergone significant transformations since the Valkey fork, driven by shifts in governance, economics, and technical advancements. Initially, Redis's licensing model change prompted the Linux Foundation to launch Valkey, a Redis fork aimed at maintaining an open-source, BSD-licensed alternative. Valkey has made notable strides, such as introducing I/O threading to improve throughput for I/O-bound workloads while maintaining a single-threaded command execution model. Despite these advancements, the real architectural innovation is seen in systems like Dragonfly, which employs a multi-threaded, shared-nothing architecture to efficiently handle the demands of AI/ML workloads. Unlike Redis and Valkey, Dragonfly is built to leverage modern multi-core, large-memory machines, offering high throughput and simplifying operations without the need for complex sharding. This makes it particularly appealing for teams focusing on AI/ML-driven applications, as it delivers consistent performance under heavy data demands without the intricate scaling challenges associated with clustered architectures. Overall, while Valkey offers a cost-effective and open-source alternative to Redis, novel solutions like Dragonfly are setting new benchmarks by addressing the shortcomings of traditional architectures in the AI/ML era.

The blog post discusses the differences in threading models between Valkey and Dragonfly, focusing on their impacts on performance in CPU-intensive workloads. Valkey 8.0 introduces enhanced asynchronous I/O threading, similar to Redis, to improve network-bound throughput, yet retains a single-threaded command execution model that limits scalability for CPU-heavy operations. This design ensures atomicity and simplicity but creates bottlenecks for tasks like sorted set operations. Dragonfly, by contrast, is built with a fully multi-threaded architecture that allows data operations to scale linearly with available CPU cores, thanks to its shard-based design and B+ tree data structures. Benchmark comparisons demonstrate that Dragonfly maintains a lower memory footprint and achieves significantly higher throughput than Valkey under the same conditions, highlighting the advantages of a multi-threaded approach for modern, compute-intensive workloads. This architectural distinction is crucial for real-world applications, particularly those requiring high computational efficiency and scalability, as Dragonfly's model allows for better performance and hardware utilization.