Streaming 500 million rows into Apache Arrow in 2.3 seconds

QuestDB reports an informal, reproducible benchmark focused on database egress performance, measuring how quickly 500 million unfiltered trade rows can be streamed to Python clients as Apache Arrow rather than how quickly they can be queried. On cache-resident data across separate AWS database and client hosts, QuestDB’s WebSocket-based QWP protocol streamed about 220 million rows per second with eight readers, delivering an initial Arrow batch in 32 milliseconds; a later reader sweep reached 266 million rows per second at 12 readers. ClickHouse’s native protocol reached roughly 142 million rows per second when run through separate processes, but buffered about 18 GB of client memory and returned no data until the full result was ready, while ClickHouse’s fastest streaming Arrow route reached about 94 million rows per second and TimescaleDB’s ADBC path about 14 million. QuestDB’s advantage in this dataset was partly attributed to its dictionary-encoded low-cardinality SYMBOL columns, which reduced network bytes per row, though the authors note that high-cardinality strings or different schemas could change the outcome. The investigation also found that early results were distorted by insufficient RAM causing disk reads, Python’s GIL constraining threaded ClickHouse native clients, and connection-coordination overhead rather than network or CPU saturation; it acknowledges limitations including a partially loaded TimescaleDB dataset, one workload and hardware configuration, and the fact that the benchmark was conducted by QuestDB’s vendor.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.