November 2023 Summaries

QuestDB, an open-source time-series database designed for high-demand workloads, has reached a significant milestone with its latest releases. The introduction of version 7.3.4 brings enhancements such as Window Functions, improved query performance, and a refreshed Web Console, while version 7.3.5 further optimizes query speeds and introduces a new percentile function. QuestDB Enterprise now offers important features like database Replication and TLS Encryption, ensuring high availability and secure data transfer for critical production environments. QuestDB Cloud integrates these updates, alongside a robust role-based access control system and an enhanced user interface that speeds up instance creation and simplifies navigation. These developments aim to provide users with more efficient and secure data management solutions, as well as a streamlined experience for both new and existing customers.

QuestDB, an open-source time-series database, leverages a specialized hash table called FastMap to optimize SQL execution, particularly for operations like JOIN and GROUP BY. FastMap differentiates itself from traditional hash tables by being designed for grow-only operations, storing data outside of the JVM's heap in native memory, and maintaining insertion order, which aids in efficient sorting during query execution. This structure uses open addressing, linear probing, and a low load factor to enhance performance, and it uniquely supports variable-size keys with fixed-size values. FastMap's design choices, including caching hash codes and employing a compact hash function, contribute to faster query execution by reducing CPU overhead and avoiding garbage collection pressures. Microbenchmarks demonstrate that FastMap performs better than the standard Java HashMap in terms of read and write operations. While FastMap is tailored for QuestDB's use cases, ongoing optimizations, such as exploring Robin Hood hashing, aim to further improve its efficiency.

QuestDB is an open-source time-series database designed for demanding workloads that provides ultra-low latency and high ingestion throughput, supporting Parquet and SQL to maintain data portability without vendor lock-in. The article explores the challenges of data deduplication, particularly in time-series and event data where duplicate entries can slow down processes and distort datasets. An experiment comparing QuestDB, Timescale, and Clickhouse reveals that QuestDB offers the most efficient deduplication with only an 8.3% performance degradation, supporting exactly-once semantics with minimal impact on ingestion performance. While Timescale ensures uniqueness through unique indexes, and Clickhouse accepts duplicates to later compact them, QuestDB achieves deduplication during ingestion using UPSERT Keys, ensuring no duplicates in query results. The native deduplication feature, introduced in QuestDB 7.3, offers strong performance while guaranteeing exactly-once semantics, making it a robust choice for applications needing reliable and efficient data handling.