Home / Companies / Mergify / Blog / Post Details
Content Deep Dive

Hypothesis is not flaky. Your code under test is, and Hypothesis is the messenger.

Blog post from Mergify

Post Details
Company
Date Published
Author
-
Word Count
1,187
Language
English
Hacker News Points
-
Summary

Hypothesis, a property-based testing tool, often reveals hidden bugs in code by generating edge cases that may not have been considered during development. A common scenario involves a test passing consistently for months but suddenly failing in a continuous integration (CI) run due to an unforeseen edge case, such as integer overflow. The text emphasizes that the failure indicates a real bug, not a flaky test, and advises against dismissing the test or modifying it to avoid the edge case. Instead, developers should capture and include the failing example as an explicit test case and fix the underlying code to handle such edge cases. This approach ensures that the bug is addressed and not ignored, as ignoring it could lead to the bug affecting production environments. The text also discusses how to use Hypothesis effectively, including strategies for reproducing failing tests by pinning seeds and differentiating between real bugs and cases that fall outside the intended scope of the function being tested. Mergify's Test Insights is highlighted as a tool that correctly identifies these failures as significant issues rather than mere flakiness, promoting a more proactive approach to testing and bug resolution.