January 2023 Summaries

QuestDB is an open-source time-series database designed for high-performance workloads, offering ultra-low latency and high ingestion throughput with a multi-tier storage engine. It supports Parquet and SQL, ensuring data remains portable and AI-ready without vendor lock-in. The article introduces a series on optimizing SQL queries using the EXPLAIN keyword, which helps users understand and improve query performance by revealing the execution plan. The story recounts the author's journey in resolving performance issues in a Java application reliant on SQL queries, highlighting the challenges faced and the eventual discovery of the EXPLAIN command as a valuable tool. QuestDB processes queries through parsing, optimization, planning, and execution phases, and EXPLAIN provides insight into these processes, showcasing how different query structures and optimizations can significantly impact performance. Examples illustrate how variations in query construction, such as using Just-In-Time compilation or index scanning, can affect response times, emphasizing the importance of understanding database fundamentals and the execution plan for enhancing performance. Despite its utility, EXPLAIN does not indicate why a specific plan is chosen, but it remains an essential tool for query performance experimentation and validation.

QuestDB is an open-source time-series database designed for high-performance workloads, offering ultra-low latency and high ingestion throughput. It supports SQL and Parquet, ensuring data portability and AI readiness without vendor lock-in. The database is particularly suitable for applications requiring real-time data processing, such as public transport systems and industrial machinery sensors, where data continuity is crucial for safety and reliability. QuestDB enhances standard SQL functionalities with extensions like SAMPLE BY, FILL, and ALIGN TO CALENDAR to facilitate the identification and handling of missing data in time-series datasets. These extensions simplify the process of finding gaps in data, allowing users to perform time-series interpolation efficiently without complex queries. The article demonstrates this capability using a trades dataset of Bitcoin and Ethereum transactions, highlighting the ease with which QuestDB can manage missing data, a task that often proves cumbersome in traditional relational databases.

QuestDB is an open-source time-series database designed for high-performance workloads, offering ultra-low latency, high ingestion throughput, and a multi-tier storage engine. It supports Parquet and SQL, ensuring data portability and readiness for AI applications without vendor lock-in. The latest release, QuestDB 6.7, introduces new features including support for the UUID data type, trigonometric functions, and an EXPLAIN SQL keyword to provide insights into query execution plans.

QuestDB is an open-source time-series database known for its ultra-low latency and high ingestion throughput, making it suitable for handling demanding workloads, such as those found in trading floors and mission control environments. It supports SQL, providing an accessible querying language for many users. The article discusses the integration of QuestDB with Prometheus using Influx Data's Telegraf as a proxy to convert Prometheus-formatted metrics into the InfluxDB Line Protocol (ILP) for ingestion. This setup allows for the storage and querying of infrastructure metrics within QuestDB, enabling complex time-series queries with SQL. The configuration involves using Helm charts for installing Telegraf and setting up remote write capabilities in Prometheus, as well as configuring QuestDB to handle metrics efficiently. Despite requiring some initial setup effort, QuestDB offers significant benefits in performance and ease of use for time-series data analysis without being confined to vendor-specific query languages.

QuestDB is a cutting-edge, open-source time-series database optimized for financial market data, offering robust ingestion throughput and SQL analytics. It is particularly well-suited for handling tick data and can be integrated with Kafka to analyze live trading activity for cryptocurrencies. A project was developed to track the prices of Bitcoin, Ethereum, and Chainlink using the public Coinbase API, with data being published to Kafka topics and stored in QuestDB via a Kafka Connect setup. The QuestDB Kafka connector utilizes the InfluxDB line protocol for efficient data ingestion and schema management. The project setup involves Docker for container management and Python scripts for data handling, allowing users to visualize data using QuestDB's UI or a REST API. The setup demonstrates how to leverage QuestDB and Kafka for real-time financial data analysis and historical trend evaluation, with further resources available for advanced visualization and configuration.

QuestDB is an open-source time-series database designed for high-demand workloads, offering ultra-low latency and high ingestion throughput with a multi-tier storage engine, supporting Parquet and SQL to ensure data portability and AI readiness. The shift from ETL to ELT paradigms allows raw data to be loaded into data lakes before transformation, which suits modern streaming architectures and is facilitated by change data capture (CDC) systems like Debezium that track and act on data changes. This setup is illustrated through a reference architecture where a Java Spring application updates stock prices in PostgreSQL, and changes are captured by Debezium, streamed to Kafka, and then ingested into QuestDB using the Kafka Connect QuestDB connector for analysis. The architecture simplifies application logic by relying on PostgreSQL for ACID-compliant transactions, Kafka for reliable data delivery, and QuestDB for storing and querying historical data. The system supports real-time and historical data analysis, as visualized through Grafana dashboards, demonstrating its capability to handle complex microservice environments by leveraging time-series queries and partitioning. This approach not only improves performance but also establishes distinct service boundaries between transactional and analytical databases, with potential extensions for long-term data storage and optimization.