Max Open Files Limit on MacOS for the JVM

QuestDB, an open-source time-series database designed for high-demand workloads, encounters specific challenges on macOS related to file descriptor limits imposed by the operating system and the Java Virtual Machine (JVM). While attempting to manage large data sets, users may hit a limit of 10240 open files, a constraint specific to the macOS JVM. This limit can be bypassed using the JVM option -XX:-MaxFDLimit, which instructs the JVM to ignore the predetermined ceiling. However, upon removing this limit, the system defaults to using the soft limit as the new threshold, which can be unexpectedly low if not adjusted. Through a series of experiments, it was demonstrated that adjusting both soft and hard limits is essential to fully leverage the -XX:-MaxFDLimit option and optimize QuestDB's performance on macOS, ensuring that system limits are not inadvertently restrictive. The mysterious 10240 limit is traced back to macOS's system configuration, specifically OPEN_MAX, which is set in Apple's system headers.