August 2023 Summaries

QuestDB is an open-source time-series database designed for high-performance workloads, offering ultra-low latency, high ingestion throughput, and a multi-tier storage engine with native support for Parquet and SQL. The database leverages a predominantly Java codebase with some C, C++, and Assembly, employing techniques to minimize garbage collection by using Zero-GC Java principles. Over time, QuestDB has integrated Rust to enhance its capabilities, especially for tasks requiring minimal garbage collection, leveraging Rust's static linking to avoid symbol collisions and its extensive library on crates.io. The integration of Rust involves creating complete components outside Java, facilitated by the development of a rust-maven-plugin to streamline the build process within the Maven ecosystem. This integration supports a seamless developer workflow in IntelliJ, enabling Rust and Java code to be compiled together, and aids in deploying QuestDB’s enterprise features like replication, cold storage, and native SSL/TLS, which benefit from Rust's performance efficiencies. The transition to Rust has been well-received within the QuestDB team, highlighting its suitability for developing scalable, enterprise-grade database features.

QuestDB is an open-source time-series database designed for high-demand workloads, offering ultra-low latency and high ingestion throughput while supporting SQL and Parquet for data portability. The text explores the complexities of designing an access control system, challenging the assumption that it is a fully resolved issue. It delves into the debate between using roles and groups to organize users, highlighting the potential for complexity and misalignment as applications evolve. The document suggests opting for single-level inheritance to maintain clarity and discusses the creation of service accounts to prevent accidental permission overlaps. Real-time enforcement of access control changes is emphasized, proposing a copy-on-write solution to ensure immediate security updates without performance issues. These innovations and refinements in access control design aim to enhance user satisfaction by minimizing complications in database interactions.

QuestDB Enterprise now offers Role-based Access Control (RBAC) to manage data access through a SQL-based syntax, addressing the growing need for secure data management in expanding datasets. This system allows administrators to efficiently assign and revoke permissions, ensuring users only have access to data relevant to their roles, such as allowing analysts like Alex to perform SELECT operations without risking data integrity. The tutorial illustrates the setup of RBAC with examples using Postgres Wire and InfluxDB Line Protocol, emphasizing the importance of security by recommending against default passwords and showcasing the flexibility of Access Control Lists (ACLs) in managing user groups and permissions. Various authentication methods, including password, JSON Web Keys (JWK), and REST API tokens, are supported, catering to different access scenarios and enhancing the security framework. Through this approach, QuestDB facilitates scalable and secure data management, allowing precise control over access permissions at multiple levels within the database system.

QuestDB, an open-source time-series database, is known for handling demanding workloads with ultra-low latency and high ingestion throughput. This article presents a detailed exploration of designing a concurrent data structure, specifically a concurrent map that favors readers by being lock-free on the read path. The author navigates the complexities of concurrent programming, including challenges such as pointer lifecycle management and consistency of map internals, while also addressing the need to integrate the map with Rust code for digital signatures verification. Through a detailed narrative, the article outlines the development of a concurrent map using techniques like Copy-On-Write and dual maps to maintain immutability and memory management efficiency. The process involves a series of trials, including the use of single-writer principles and reader counters, to ensure that readers and writers do not access the same map concurrently. The implementation, while effective, is recognized as still needing further development for production readiness, such as implementing a close method to prevent memory leaks and optimizing performance. The article acknowledges contributions from existing patterns and papers, highlighting the iterative learning and adaptation involved in concurrent programming design.

QuestDB, an open-source time-series database designed for high-performance workloads, has significantly improved its robustness by incorporating fuzz testing, an automated software testing technique that introduces unexpected or random data inputs to uncover bugs. Initially faced with challenges such as segfaults, data corruption, and concurrency issues, the QuestDB team implemented fuzz tests to address these problems, leading to a more stable database and a decrease in critical issues reported by the community. The fuzz tests, which include approaches like generation-based and mutation-based testing, have helped identify numerous bugs by applying randomness to various scenarios, especially in relation to the InfluxDB Line Protocol and SQL engine. The team also benefited from SQLancer, a tool for identifying logic bugs in SQL Database Management Systems, which reinforced their testing efforts. Through these methods, QuestDB has not only enhanced its own system but has also advocated for the use of fuzz testing across complex software applications to improve reliability and performance.

QuestDB 7.3 introduces significant updates, including data deduplication and IPv4 support, enhancing data management and query performance. Data deduplication helps maintain data clarity by eliminating duplicates before they are stored, thus saving storage costs and improving SQL query efficiency without affecting ingestion performance. This is particularly beneficial for time-series data with frequent overlaps. The release also supports IPv4 addresses, allowing for more efficient IP tracking and management with operations such as bitwise functions and subnet checks, while being stored as integers for better performance. Additionally, QuestDB 7.3 includes a new SQL syntax for combining two long integers into a unique UUID, improved tuning for Write-Ahead Log (WAL) tables, and various fixes and stability enhancements. The update does not introduce any breaking changes, ensuring compatibility with existing workflows, and offers comprehensive documentation for these new features.