Repeatable Performance Tests: EC2 Instances are Neither Good Nor Bad

EC2 instances were initially assumed to have good or bad performance based on the cluster they were run on, but a series of experiments and data analysis revealed that there is no correlation between test results and the clusters. The variance in performance does not correlate with the clusters, suggesting that EC2 instances are neither inherently good nor bad, and that other factors such as configuration, instance type, and usage patterns may be more significant contributors to variability. By measuring their system and analyzing data in different ways, the MongoDB Performance team was able to disprove one of their assumptions and gain a better understanding of what contributes to performance variability on EC2 instances.