Company
Date Published
Author
Carlos Bueno
Word count
604
Language
English
Hacker News points
None

Summary

The SingleStore database has introduced a new feature to provide detailed statistics about memory use, which was a frequent request from users. The `information_schema` table now exposes this information, allowing users to roll up the data to generate totals such as the size of a table in the cluster. However, skew can occur when rows are sharded according to specific columns, leading to uneven distribution of data across machines. To measure skew, the relative standard deviation is used, with values below 0.05 being considered acceptable. The new feature was developed through collaboration between the "engine" team building the database and the "ops" team building monitoring apps, resulting in a better feedback loop and more efficient software development process. This allows for faster prototyping of visualizations using real data, while also informing improvements to the underlying database features.