Repeatable Performance Tests: CPU Options Are Best Disabled

The MongoDB team conducted an experiment to improve the repeatability of performance tests on EC2 instances by investigating CPU options and their impact on noise in system performance. They recorded data about their current setup, found that EBS instances were stable for IO operations, and then focused on tuning CPU-related knobs to minimize noise. The investigation involved testing various CPU options, including idle strategy, kernel boot, CPU frequency, and hyperthreading, which revealed that disabling these features resulted in lower noise levels. The team also ran tests with different configurations, comparing results across client-only runs and connected server runs, finding that the c3.8xlarge instance configuration yielded consistently low noise levels when using all tuning options. Furthermore, they discovered that CPU tuning had a significant impact on IO-heavy tests, which were previously thought to be less affected by CPU tuning. The experiment demonstrated the importance of fine-tuning CPU options for achieving stable and repeatable results in performance testing.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.