Company
Date Published
Author
Alexey Milovidov
Word count
684
Language
English
Hacker News points
1

Summary

ClickHouse has a robust testing infrastructure, with over 1.7 billion test records collected since June 2020. The team runs around 5-9 million tests daily, which helps ensure the stability and quality of the software. While some tests can be flaky due to various reasons such as timing issues or subtle logic errors, the ClickHouse team takes a different approach to deal with these tests. Instead of ignoring them, they intentionally make their tests more flaky by randomizing environments, using fuzzing, and adding retries to investigate and fix the causes of failure. The team visualizes flaky tests using a "Christmas Tree" visualization, which shows the success or failure of individual tests over time, helping identify trends and areas for improvement. This approach allows the team to proactively address issues and maintain a high-quality testing infrastructure.