May 2023 Summaries

QuestDB, an open-source database designed for market data, excels in high ingestion throughput and SQL analytics, making it ideal for analyzing tick data. Paired with Pandas, a popular Python library for data manipulation, it provides a robust tech stack for handling and analyzing time-series data. This combination allows users to leverage Pandas' in-memory data structures for preliminary exploration and QuestDB's efficient storage for large datasets, mitigating Pandas' memory limitations. The article demonstrates this synergy through a tutorial on analyzing historical crypto prices, utilizing various methods to ingest data into Python or QuestDB, and performing analyses with Pandas or SQL to visualize results in Jupyter Notebook. The workflow's effectiveness depends on data size, format, and team structure, offering flexibility as data or team dynamics evolve.

QuestDB, an open-source time-series database known for its ultra-low latency and high ingestion throughput, used the Time-Series Benchmark Suite (TSBS) to optimize its performance. Initially based on InfluxDB's benchmarking tool, QuestDB's team discovered that TSBS itself was a bottleneck, prompting them to create a fork of TSBS for better optimization. By profiling and modifying the TSBS code, they improved data ingestion rates significantly, achieving up to 4.3 million rows per second. These enhancements included relaxed data validation and the removal of flow control in the TSBS loader. Despite submitting pull requests to the original TSBS repository, the lack of response led QuestDB to maintain their own fork, which includes optimizations, bug fixes, and support for InfluxDB v2. QuestDB continues to invite contributions to their open-source projects while promoting community engagement through their forum and live demo.

QuestDB, an open-source time-series database designed for high-performance workloads, encountered an unexpected issue involving disk reads during a write-only workload. This anomaly was traced back to the Linux kernel's readahead behavior, which led to unnecessary disk read operations due to memory pressure from handling a large number of column files. The investigation involved using Linux utilities like blktrace and debugfs to track disk read events and identify their source. The problem was resolved by disabling readahead using the madvise system call, which prevented redundant reads. This case highlighted the importance of understanding Linux's buffered I/O and using system-level tools to address performance issues, ultimately leading to improvements in QuestDB's functionality. The experience reinforced the value of user feedback in enhancing the database's performance and functionality.