Building a new vector based storage model

QuestDB is an open-source time-series database designed for high-performance workloads, offering ultra-low latency and high ingestion throughput with a multi-tier storage engine. Initially inspired by vector-based, append-only systems like kdb+, QuestDB struggled with handling out-of-order (O3) data, a common occurrence in real-world applications due to network issues and clock synchronization. The database previously discarded out-of-order records due to its data model's reliance on ordered timestamps, prompting the development of a new solution in QuestDB 6.0. This solution introduced a three-stage process involving appending data, sorting uncommitted records in-memory, and reconciling sorted O3 data during commit time, effectively integrating O3 support without sacrificing performance. This approach employed SIMD and non-temporal data access for efficient data handling, while also exploring the potential of O3 hysteresis to optimize performance further. Through rigorous benchmarking and optimization, QuestDB achieved a novel and efficient method to support O3 data, distinguishing itself from other databases using B-trees or LSM trees.