July 2025 Summaries

QuestDB, an open-source time-series database tailored for demanding workloads, offers advanced features for handling Level 2 (L2) data, which is crucial for understanding market depth in trading and financial markets. Unlike traditional relational databases that struggle with the efficient storage and querying of L2 data due to schema complexity and performance bottlenecks, QuestDB's support for float64 arrays from version 9.0.0 onwards allows for streamlined data ingestion and querying. The tutorial demonstrates how to modify Cryptofeed to store L2 data as arrays, showcasing the setup and execution of a Python environment using Docker to run QuestDB and ingest data from exchanges like Bitstamp. It further illustrates querying capabilities, such as calculating average bids and asks and determining price levels for specific volumes using array-specific functions. The text highlights the compactness and efficiency gained by using arrays for data representation, which facilitate better analysis and trend identification, and also explores alternative order book layouts utilizing bi-dimensional arrays for more complex data structures.

QuestDB's version 9 introduces N-dimensional arrays as a pivotal data type, akin to storing a NumPy array within a single database column, enhancing the efficiency of processing financial market data with vectorized SIMD instructions. This innovation simplifies the storage and analysis of order books, traditionally managed with multiple columns, by representing them as compact two-dimensional arrays, specifically for order book analytics where prices and volumes are stored in rows. This approach facilitates more streamlined and powerful analytics, such as calculating bid-ask spreads, total liquidity, and market impact analyses, all while optimizing performance and reducing query complexity. The blog post provides various examples of how these arrays can be used for market analysis, including liquidity concentration, order book imbalance, and spread monitoring, demonstrating the potential for sophisticated market microstructure analysis. By integrating time-series capabilities with these array operations, QuestDB offers a robust platform for exploring market dynamics, encouraging users to leverage this new feature for deeper insights into financial data.

QuestDB is an open-source time-series database designed for high-performance workloads, offering features like ultra-low latency, high ingestion throughput, and a multi-tier storage engine with native support for Parquet and SQL. A key feature of QuestDB is its ASOF JOIN functionality, which is used to join time-series data such as matching trades to quotes based on timestamps. Traditionally, ASOF JOIN matches records from one table with the latest record from another table that occurred at or before the timestamp, which could sometimes result in joins with outdated data. To address this, QuestDB introduced the TOLERANCE clause, allowing users to specify a maximum time window for the join, ensuring that matches are not only the latest but also relevantly recent. This enhancement provides finer control over the data, preventing misleading joins with stale data, and can improve query performance by reducing unnecessary data scanning. The TOLERANCE clause is particularly advantageous in environments where data is produced or consumed rapidly, such as financial markets or real-time monitoring, where the recency of information is crucial for accurate analysis.

QuestDB 9.0 introduces significant enhancements to the open-source time-series database, focusing on performance and usability for demanding workloads. Key updates include the addition of N-dimensional arrays, which allow for efficient operations similar to NumPy arrays, and improvements in materialized views with new refresh modes that accommodate global time zones. Enhanced data deduplication mechanisms minimize I/O overhead, and the web console has been revamped for better user experience, including features like per-line query runners and a new query log. The ASOF JOIN feature now supports a TOLERANCE parameter for more precise time-series data matching. This release aims to bolster QuestDB's capabilities in high-throughput environments, with the promise of future updates such as full Parquet support and nanosecond timestamps.