To trust an LLM, make lying harder than telling the truth

At Swimm, the challenge of generating reliable functional specifications for complex applications involves a delicate balance between deterministic methods and large language models (LLMs), such as GPT-4.1. The company discovered the limitations and pitfalls of relying on LLMs, particularly their tendencies to lie, hallucinate, or forget when tasked with validating semantic equivalence between specs. Through a series of experimental attempts, the Swimm team developed a structured approach that forces LLMs to provide evidence before making conclusions, thus minimizing the risk of incorrect matches and hallucinations. This method involved extracting requirements, cross-checking them, and requiring detailed evidence for matches, which significantly improved the reliability of the specs. The experience underscored the importance of structured outputs and specific requirements to ensure accountability and truthfulness in AI-generated documentation, highlighting a broader lesson about designing interfaces that discourage deception by LLMs. This work is part of Swimm's ongoing effort to build trustworthy AI-driven documentation for complex codebases, leveraging Omer Rosenbaum's expertise as CTO and Co-founder.