Small Changes for Big SQLite Performance Increases

The WhyLabs engineering team improved SQLite performance by making monitoring data and machine learning models faster and easier for whylogs users. They increased transactions per second (TPS) and p95 latency by orders of magnitude through some small, obscure changes to the SQLite code. These improvements included omitting vacuums, reusing a single database connection for the life of the container, handling disabling autocommits, and committing at the end of blocks. The new code performs significantly better than the old one, with roughly 5 times faster TPS and a p95 request time that is consistently fast enough for users to monitor their data & machine learning models easily and quickly.