AI Brittleness vs. Non-Determinism: The Real Reliability Problem

The text discusses the challenges of ensuring reliability in AI systems, particularly in distinguishing between non-determinism and brittleness. Non-determinism is when identical inputs produce different outputs due to factors like stochastic sampling, which can be controlled with engineering techniques such as temperature settings and fixed seeds. Brittleness, however, arises when semantically equivalent inputs yield different outputs due to slight variations in phrasing, which is not addressed by controlling temperature. The text highlights the importance of identifying brittleness through methods like paraphrase testing and adversarial input variation, as well as the cost implications of conflating these issues. It emphasizes that production-ready AI must demonstrate stable behavior across a range of real-world input variations, rather than just achieving high accuracy on clean test sets, to avoid costly errors and ensure reliability.