August 2020 Summaries

QuestDB is an open-source time-series database designed for high-performance applications like trading floors and mission control, offering ultra-low latency, high ingestion throughput, and a multi-tier storage engine with native Parquet and SQL support. Shan Desai, a software engineer at Emerson Discrete Automation, provides a tutorial on deploying QuestDB for IoT applications using a repository called "Questitto," which simplifies the setup of a stack involving Telegraf, InfluxDB, Grafana, and Mosquitto MQTT Broker. The tutorial includes details on user authentication, data ingestion, and configuration using Docker, emphasizing the use of SQL queries and dynamic timestamping features. It demonstrates how to extract sensor data using MQTT topics and manage data insertion into QuestDB without needing a predefined schema. Additionally, it highlights the ease of visualizing data through QuestDB's UI and offers tips for managing configurations without disrupting the stack.

QuestDB is an open-source time-series database engineered for high-performance workloads, offering ultra-low latency, high ingestion throughput, and a multi-tier storage engine with native support for Parquet and SQL to ensure data portability and AI readiness. A recent code modification, initially anticipated to degrade performance, surprisingly resulted in a significant boost, demonstrating the team's ongoing learning and adaptation process. Initially, QuestDB utilized a single-threaded approach for queries, but performance improvements were achieved through parallelization via multi-threading. The database stores data in a columnar format, with each column in a separate file, split into independent data frames. However, handling variable-length data within these frames posed challenges, leading to inefficiencies. To overcome this, QuestDB adopted a strategy of using a single, large memory-mapped page for data storage, which eliminated the inefficiencies of jumping between pages and resulted in a 60% performance improvement. The operating system's kernel efficiently handles paging, allowing QuestDB to leverage its capabilities without attempting to override them. Performance benchmarks demonstrated QuestDB's superior speed, especially compared to other databases like Postgres, highlighting the importance of utilizing the kernel's inherent efficiencies.

QuestDB is an open-source time-series database designed for high-demand workloads, offering ultra-low latency and high ingestion throughput, making it suitable for applications like trading floors and mission control. The database supports Parquet and SQL, ensuring data portability and AI readiness without vendor lock-in. The founder's experience began in 2012 while working at an energy trading company, where he discovered the limitations of existing databases and explored memory-mapping techniques in Java. This led to the creation of a custom storage engine that used off-heap memory and a query system for geospatial time series. Over the years, the founder left his job to focus on developing QuestDB full-time, emphasizing performance optimization and avoiding garbage collection by rewriting standard libraries. Despite initial setbacks due to design flaws, a complete rewrite improved the database's performance and capabilities, enabling atomic and durable multi-column commits and optimized CPU use. The project gained traction as contributors joined, further advancing QuestDB's development and expanding its user base.