May 2024 Summaries

QuestDB is an open-source time-series database known for its ultra-low latency and high ingestion throughput, making it ideal for demanding workloads such as trading floors and mission control. The recent release of QuestDB 8.0 introduces significant performance enhancements, including an average of 50% faster query speeds due to multiple optimizations. A major update is the availability of system-level compression for open-source users, expected to reduce storage utilization by at least 6x, and the introduction of the VARCHAR data type, which optimizes storage and query efficiency compared to the older STRING type. This version also includes finance-specific functions like the Level-2 Price function, which is essential for high-speed trading by providing detailed market data. QuestDB 8.0's development was supported by a collaborative community effort, inviting feedback and engagement through their Community Forum.

This article details the process of upgrading a Raspberry Pi with an M.2 NVMe SSD to enhance its performance, and subsequently setting up QuestDB, an open-source time-series database known for its high ingestion throughput and low latency. It outlines the steps for physically attaching the NVMe drive to the Raspberry Pi using a Pimoroni M.2 extension kit, configuring the Raspberry Pi firmware to recognize the new drive, and installing the Ubuntu Server operating system on it. The article further explains the setup of a benchmarking suite and QuestDB on the Raspberry Pi, including the installation of necessary software like Go, the Time Series Benchmarking Suite (TSBS), and Java, along with compiling and configuring QuestDB to prepare for benchmarking tasks. The setup is designed to optimize data movement by utilizing the NVMe's speed, which significantly surpasses that of standard SD cards, and invites readers to engage in benchmarking and share their results through social media or community forums.

QuestDB serves as an open-source time-series database tailored for demanding applications, providing ultra-low latency and high ingestion throughput, with native support for Parquet and SQL to ensure data portability. The tutorial guides users on creating a resource monitor that polls system resource usage, stores the information in QuestDB, and visualizes it using Grafana in real-time. Utilizing the Python library psutil, the tutorial illustrates how to monitor and manage system processes by collecting data on CPU, RAM, and other system metrics. The guide demonstrates correlating system resource data with application events using SQL queries and explains how to enhance data visualization and readability with Grafana's transformation features. By the end, users gain a foundational understanding of resource monitoring and event tracking, with the potential to build scalable monitoring systems leveraging QuestDB and Grafana.