Company
Date Published
Author
Bojan Serafimov
Word count
2241
Language
English
Hacker News points
12

Summary

Neon, a serverless Postgres system, needs to solve the problem of searching for WAL (Write-Ahead Log) records in its storage system. The read path involves reconstructing requested pages by applying WAL records to recent page images. To speed up this process, Neon introduces a new data structure that uses persistent data structures (RPDS) from the Rust crate to achieve a 70x to 2000x speedup compared to previous non-trivial data structures. The RPDS data structure is based on binary search trees and allows for efficient querying of past states by traversing layers, which are essentially snapshots of the data at different points in time. This approach enables Neon to efficiently answer historical queries with a query runtime of O(log N), making it suitable for large-scale databases.