Why LLMs Can't Really Build Software

Interviewing software engineers has provided insights into the key abilities that distinguish effective practitioners, particularly their capacity to build and maintain clear mental models throughout the software engineering loop. While large language models (LLMs) can assist in writing and updating code, they fall short in maintaining coherent mental models, often struggling with context omission, recency bias, and hallucination. Unlike human engineers, LLMs cannot yet manage the complexity of iterative problem-solving due to these limitations. Despite their current constraints, LLMs are valuable tools for generating code and synthesizing requirements, but human engineers remain crucial for ensuring accuracy and clarity in software development. At Zed, there is a belief in collaborative efforts between humans and AI, with humans taking the lead in driving the development process.