Load testing Grafana k6: Peak, spike, and soak tests

Grafana Labs conducted a series of unconventional load tests using Grafana k6 to explore the performance limits of their load generation system, focusing on the system's ability to handle high volumes of data and high-cardinality metrics. Unlike typical load tests that concentrate on the system under test (SUT), the focus here was on the load generator's capacity, which involved atypical design choices like using an invincible SUT and generating unrealistically large loads. The tests—peak, spike, and soak—revealed insights into system bottlenecks, particularly in storage and database locking, and highlighted the need for better aggregation and autoscaling strategies. Grafana Labs emphasized the importance of monitoring alongside testing to understand system health and user experience, and noted that different test types can uncover distinct system weaknesses. The findings reinforced the notion that load testing must be tailored to each system's unique characteristics and that the real best practice involves thoughtful, system-specific test design.