In-memory vs. disk-based databases: Why do you need a larger than memory architecture?

Memgraph has introduced a larger-than-memory architecture to its in-memory graph database, allowing users to store data that exceeds available RAM on disk while maintaining the benefits of in-memory computation. This approach addresses the limitations of traditional in-memory databases, which struggle with data exceeding memory capacity, and disk-based databases, which are slower due to disk access times. By utilizing techniques like identifying hot and cold data and employing RocksDB for disk storage, Memgraph aims to provide efficient data management without significant performance loss. The architecture implements Optimistic Concurrency Control and snapshot isolation for transaction management, ensuring minimal impact on the original in-memory performance despite the complexity of integrating disk storage. However, challenges remain, such as the requirement for transaction data to fit into memory, which limits the execution of large analytical queries and affects certain functionalities like the LOAD CSV command. Memgraph continues to address these issues to enhance the larger-than-memory architecture's utility for users.