November 2022 Summaries

QuestDB is an open-source time-series database designed for high-demand workloads, offering low latency, high ingestion throughput, and a multi-tier storage engine. It supports Parquet and SQL, ensuring data portability without vendor lock-in. Jaromir Hamala, one of QuestDB's core engineers, conducted performance tests inspired by a study comparing index merging and composite indexes, particularly focusing on QuestDB's table scanning strategy. His findings highlighted the advantages of table scanning in handling complex analytical queries, demonstrating that QuestDB could perform a full table scan of 10 million rows in under 5 milliseconds, rivaling the speed of PostgreSQL's composite indexes and outperforming MySQL's index merges. Although indexes are not deemed useless, as they can provide selectivity and performance improvements in certain scenarios, QuestDB's reliance on table scans leverages modern hardware's capabilities effectively, making it suitable for ad-hoc queries common in analytics. This approach is exemplified in a public demo where QuestDB efficiently aggregates large datasets, showcasing its focus on maximizing raw performance through optimal use of hardware.

QuestDB is an open-source time-series database designed for high-demand environments, offering features like ultra-low latency, high ingestion throughput, and a multi-tier storage engine with native support for Parquet and SQL. The release of QuestDB 6.6.1 introduces dynamic commits, which enhance data ingestion throughput and freshness by optimizing the commit lag—a buffer that helps reorder out-of-order data in memory before committing it to disk. Previous versions required users to manually configure the commit lag based on their data patterns, often leading to frustration when data was not immediately available for reads. The new dynamic commit system automatically adjusts the commit lag value based on data overlap patterns, thus improving user experience by ensuring data is quickly available without the need for complex configurations. This development, driven by community feedback and internal collaboration, allows QuestDB to better handle diverse data ingestion methods while maintaining optimal data immediacy.

QuestDB, an open-source time-series database, is designed for high-performance workloads and supports SQL and Parquet, ensuring data portability and AI readiness. This tutorial focuses on the SAMPLE BY SQL extension, which simplifies the grouping and aggregation of time-series data by designated timestamps, eliminating the need for complex SQL statements. By using SAMPLE BY, users can efficiently handle issues like missing data, incorrect time zones, and offsets. The extension allows for aggregation across various time units, from microseconds to years, and supports filling missing data through linear interpolation or other methods. Additionally, SAMPLE BY helps align timestamps to different time zones or offsets, enhancing data consistency across diverse systems. The tutorial provides practical examples using NYC taxi rides data from February 2018, demonstrating how to implement these features in QuestDB for efficient time-series data analysis.

QuestDB, an open-source time-series database known for its high performance and open standards, recently participated as a Silver sponsor at Devoxx Belgium, a prominent tech event for the Java community. The event, held in Antwerp in October 2022, provided a platform for QuestDB to showcase its capabilities, including its ability to ingest over 1.5 million records per second, and its fully open-source nature under the Apache 2.0 license. CTO Vlad's presentation on improving CSV import speed using io_uring was part of the Server Side Java track, attracting interest from attendees across diverse sectors such as finance, industrial IoT, and telecommunications. QuestDB engaged with over 100 attendees, offering hands-on experiences with their demo datasets, which helped them gather valuable insights into user needs and potential improvements for their database solutions.

QuestDB is an open-source time-series database designed for high-performance workloads, such as those on trading floors and mission control, offering ultra-low latency, high ingestion throughput, and a multi-tier storage engine. It supports Parquet and SQL, ensuring data portability and AI-readiness without vendor lock-in. QuestDB addresses the challenge of storing and managing vast amounts of time-series data, often required for real-time anomaly detection and long-term forecasting, by providing efficient downsampling and partition management features to maintain operational efficiency and prevent database bloat. The database allows for the easy detachment or dropping of partitions to manage data retention, enabling users to maintain a lean database by moving older data to more affordable storage solutions, with the flexibility to reattach or permanently delete partitions as needed. This flexibility is illustrated through an IoT application managing heart rate data, demonstrating how QuestDB can efficiently handle continuous data streams, downsample data for trend analysis, and manage historical data storage, ensuring that data science teams can focus on analysis without being burdened by massive raw datasets.