Importing 300k rows/sec with io_uring
Blog post from QuestDB
QuestDB's latest update, version 6.5, introduces a highly optimized CSV file import feature using the SQL COPY statement, significantly enhancing the speed and efficiency of data ingestion. Leveraging io_uring, a modern asynchronous I/O interface, QuestDB has improved its import performance, surpassing several well-known OLAP and time-series databases in the ClickBench benchmark, with only ClickHouse achieving faster results. The new import method, which processes data in parallel through multiple phases, has reduced import times dramatically, particularly when using high-memory setups or NVMe SSDs. This update allows for in-order data ingestion and reduces the need for complex partition re-writes, optimizing data layout on disk for immediate querying. The improvements in QuestDB's import capabilities were driven by insights gained from the ClickBench benchmark, which also highlighted opportunities for enhancing query stability and performance. The update positions QuestDB closer to a practical alternative to ClickHouse for time-series data analysis and sets the stage for further innovations, including potential applications of io_uring in QuestDB Cloud.