GlareDB Is Getting a Whole New Engine

GlareDB is undergoing a significant transformation with the development of a new custom execution engine, moving away from DataFusion to better support complex SQL queries and enhance both ergonomic and reliable analytics. The redesign focuses on accommodating intricate queries such as deeply nested subqueries and complex joins, ensuring users can efficiently obtain insights without the system crashing under pressure. This shift involves meticulous resource management, including buffer allocation and memory usage, inspired by Arrow's columnar layout but optimized for performance. Although support for external systems like BigQuery and Postgres has been temporarily removed, GlareDB continues to prioritize integration with file formats such as CSV, Parquet, and JSON, while investing in compatibility with Iceberg and Delta Lake. Future updates will provide more technical details and opportunities for users to test the new engine on practical workloads.