June 2023 Summaries

QuestDB 7.2 introduces implicit variable-size time partitions to enhance performance by reducing write amplification, especially during heavy loads. This new feature strategically splits partitions to maintain smaller sizes, ensuring high write performance and minimizing the need for copy-on-write operations. Previously, fixed-size time partitions led to increased write amplification towards the end of each time interval. The update also includes a sophisticated statistical model to optimize partition management, asynchronously squashing partitions unlikely to be modified, thus reducing disk files and easing the OS metadata catalog. This release includes breaking changes, as partitions created in 7.2 may not be compatible with earlier versions, requiring users to potentially squash partitions when downgrading.

QuestDB, an open-source time-series database designed for high-demand workloads, encounters specific challenges on macOS related to file descriptor limits imposed by the operating system and the Java Virtual Machine (JVM). While attempting to manage large data sets, users may hit a limit of 10240 open files, a constraint specific to the macOS JVM. This limit can be bypassed using the JVM option -XX:-MaxFDLimit, which instructs the JVM to ignore the predetermined ceiling. However, upon removing this limit, the system defaults to using the soft limit as the new threshold, which can be unexpectedly low if not adjusted. Through a series of experiments, it was demonstrated that adjusting both soft and hard limits is essential to fully leverage the -XX:-MaxFDLimit option and optimize QuestDB's performance on macOS, ensuring that system limits are not inadvertently restrictive. The mysterious 10240 limit is traced back to macOS's system configuration, specifically OPEN_MAX, which is set in Apple's system headers.