March 2023 Summaries

QuestDB is an open-source, next-generation database optimized for market data, known for its high ingestion throughput, advanced SQL analytics, and hardware efficiency, making it suitable for handling tick data. Apache Kafka serves as a robust distributed stream-processing platform ideal for managing real-time market data, crucial for trading, risk management, and fraud detection in financial institutions. A typical data pipeline involves streaming data from market feeds to Kafka and then storing it in databases using Kafka Connect. This article demonstrates creating a sample data pipeline that polls real-time stock and ETF quotes from FinnHub, publishes them to Kafka, and utilizes Kafka Connect to stream this data into QuestDB for analysis. The setup involves building a Docker environment with Kafka, Kafka Connect, and QuestDB, and configuring Kafka Connect sinks for specific stocks like Tesla and SPY. The process includes setting up prerequisites such as Git, Docker, Golang, and a FinnHub API token, and executing a Golang script to retrieve and publish stock prices to Kafka topics every 30 seconds. The data is then accessible in QuestDB for further querying and analysis, illustrating the pipeline's capability to handle real-time financial data effectively.

QuestDB, an open-source time-series database, is designed for high-demand workloads, offering ultra-low latency, high ingestion throughput, and multi-tier storage with native Parquet and SQL support to ensure data portability and AI readiness. The database, built with C++, Java, and Rust, aims to implement high availability through replication, similar to the challenges faced in maintaining redundancy in everyday tasks like biking. The text explores different replication strategies across popular databases, detailing how QuestDB plans to integrate multi-master replication with asynchronous consistency to balance fault tolerance and transaction throughput. By using a Write Ahead Log (WAL), QuestDB enhances parallelism, achieving significant improvements in write performance. The database plans further development to distribute the Sequencer component and enable WAL sharing among multiple instances, while inviting community engagement through forums and open-source contributions.

QuestDB provides insights into the unique characteristics and benefits of time-series data, highlighting key considerations for migrating from a relational database to a time-series database. This transition can optimize data management and improve performance for time-sensitive applications by leveraging the specialized features of time-series databases, such as efficient data storage and retrieval for sequential data points. QuestDB offers a deeper understanding of these databases' capabilities, which can be particularly advantageous for businesses dealing with high-frequency data and requiring real-time analytics.

QuestDB is an open-source time-series database designed for high-performance workloads, offering ultra-low latency, high ingestion throughput, and a multi-tier storage engine, with native support for Parquet and SQL to ensure data portability. In a comparison with MongoDB, a popular NoSQL document-oriented database, QuestDB demonstrates significantly faster ingestion speeds, completing tasks 24 times quicker than MongoDB, while also excelling in complex query performance due to its columnar data layout, SIMD instructions, and multi-threaded processing. The evaluation highlights the ease of use with QuestDB's SQL-based query language, which simplifies data aggregation tasks compared to MongoDB's JavaScript-based MongoDB Query Language (MQL). Both databases have strengths in their ecosystems and support, with MongoDB being a more mature product with a vast ecosystem and QuestDB offering vibrant community engagement and compatibility with the Postgres wire protocol. While MongoDB has added time-series functionalities, QuestDB's purpose-built design for time-series data provides unique advantages, such as efficient data lifecycle management and simpler query syntax for time-based data analysis.

QuestDB is an open-source time-series database designed for high-demand workloads, offering ultra-low latency, high ingestion throughput, and a multi-tier storage engine, with native support for Parquet and SQL to keep data portable and AI-ready. While the Pandas library is essential for data scientists working with Python due to its intuitive data manipulation capabilities, it struggles with large datasets that exceed available machine memory. By integrating Pandas DataFrames with QuestDB, users can leverage the database's robust data processing capabilities to efficiently handle and analyze large datasets. This tutorial demonstrates how to load NYC taxi trip records into QuestDB using Docker and Python, overcoming memory constraints by ingesting data one file at a time through QuestDB’s Python client. The method allows for scalable data analysis and manipulation, offering insights such as average passenger payments, and can be customized for specific needs, highlighting the potential of combining Pandas and QuestDB for big data tasks.

Kelsey Hightower's discussion on running databases on Kubernetes highlights the challenges and misconceptions associated with using Kubernetes for stateful workloads such as databases. While Kubernetes offers cloud-native primitives like StatefulSets, PersistentVolumes, and PersistentVolumeClaims to support stateful applications, these are merely building blocks that require additional orchestration for full functionality. The discussion emphasizes that Kubernetes does not automatically provide complex lifecycle features typical of managed database services, leading to the necessity for Day 2 Operations, which include tasks like backups, restores, replication, and high availability. Despite its complexities, Kubernetes offers benefits such as programmatic workload management and multi-cloud flexibility, making it appealing for organizations willing to invest in the expertise needed to implement these advanced operations. The text elaborates on how Kubernetes' operator model and tools like Helm can support database automation, though they require significant effort and expertise to implement effectively. Ultimately, the choice to run databases on Kubernetes should be informed by the specific needs and resources of an organization, as it can offer great flexibility and control but demands a high level of technical investment and understanding.

QuestDB, an open-source time-series database designed for high-demand environments, recently faced a series of build failures attributed to unstable connections in their Azure Pipelines setup. The issue was traced to the Maven 3.9 update, which defaulted to a new "Native" transport protocol lacking retry capabilities, unlike its predecessor, the "Wagon" transport. This change led to consistent build failures due to terminated connections during long test runs. Through detailed troubleshooting, including examining Maven's source code, a temporary fix was found by reverting to the "Wagon" transport using a specific build command flag. This discovery was reported to Apache Maven, leading to collaborative efforts with maintainers to address the issue. The experience underscored the significance of reading release notes, the value of open-source contributions, and gratitude towards the maintainers of essential software infrastructure.