Dosu is an AI teammate that helps develop, maintain, and support software projects by taking tasks off developers' plates. It was born out of the need to address burnout among open source software maintainers, who spend more time playing support than developing new features. Dosu uses evaluation driven development (EDD) to ship with confidence, monitoring and searching its activity to ensure reliability. With the growth of Dosu, it became challenging to monitor responses and identify failure modes in production, prompting the use of LangSmith, a tool that provides visibility into all of Dosu's activity, advanced search functionality, and customizable metadata tracking. By leveraging LangSmith, Dosu can identify failure modes, integrate it into its EDD workflow, and automate evaluation dataset collection, ultimately speeding up the development process.