Re-examining our approach to memory mapping

QuestDB is an open-source time-series database engineered for high-performance workloads, offering ultra-low latency, high ingestion throughput, and a multi-tier storage engine with native support for Parquet and SQL to ensure data portability and AI readiness. A recent code modification, initially anticipated to degrade performance, surprisingly resulted in a significant boost, demonstrating the team's ongoing learning and adaptation process. Initially, QuestDB utilized a single-threaded approach for queries, but performance improvements were achieved through parallelization via multi-threading. The database stores data in a columnar format, with each column in a separate file, split into independent data frames. However, handling variable-length data within these frames posed challenges, leading to inefficiencies. To overcome this, QuestDB adopted a strategy of using a single, large memory-mapped page for data storage, which eliminated the inefficiencies of jumping between pages and resulted in a 60% performance improvement. The operating system's kernel efficiently handles paging, allowing QuestDB to leverage its capabilities without attempting to override them. Performance benchmarks demonstrated QuestDB's superior speed, especially compared to other databases like Postgres, highlighting the importance of utilizing the kernel's inherent efficiencies.