Reducing Variability in Performance Tests on EC2: Setup and Key Results

The MongoDB team set out to reduce variability in performance tests on EC2 by analyzing and controlling for environmental noise in their test configuration. They used a scientific approach to measure and analyze the noise, applying the "Assume nothing. Measure everything." principle to their experiments. By repeating tests multiple times across different EC2 clusters, they were able to identify factors contributing to variability, such as disk performance, CPU tuning, and instance types. The team found that using Provisioned IOPS and EBS disks significantly reduced test results' variation, while disabling CPU options also stabilized performance. They established canary benchmarks to monitor EC2 performance and detect potential issues before they affect MongoDB's performance. By reducing system noise, the team improved the reliability of their System Performance tests, enabling them to detect changes in MongoDB's performance, diagnose issues promptly, and address them before releases.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.