Building a new vector based storage model

QuestDB is an open-source time-series database designed for high-performance workloads, offering ultra-low latency and high ingestion throughput with a multi-tier storage engine. Initially inspired by vector-based, append-only systems like kdb+, QuestDB struggled with handling out-of-order (O3) data, a common occurrence in real-world applications due to network issues and clock synchronization. The database previously discarded out-of-order records due to its data model's reliance on ordered timestamps, prompting the development of a new solution in QuestDB 6.0. This solution introduced a three-stage process involving appending data, sorting uncommitted records in-memory, and reconciling sorted O3 data during commit time, effectively integrating O3 support without sacrificing performance. This approach employed SIMD and non-temporal data access for efficient data handling, while also exploring the potential of O3 hysteresis to optimize performance further. Through rigorous benchmarking and optimization, QuestDB achieved a novel and efficient method to support O3 data, distinguishing itself from other databases using B-trees or LSM trees.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.