CommandBar is a user assistance platform designed to enhance software usability through its Copilot widget, which predicts user intent and delivers personalized in-product assistance. This tool, embedded in client applications, can address user queries, initiate tailored product tours, and even execute tasks on users' behalf. To ensure the Copilot's effectiveness across diverse clients, CommandBar integrated large language models (LLMs) and multiple content providers, leveraging LangSmith to monitor and improve Copilot's performance. LangSmith facilitated trace visibility, debugging, increased testing coverage, and monitoring, proving instrumental in refining the Copilot's capabilities. Since its deployment in November 2023, Copilot has significantly reduced customer support tickets by 44% and has become a key competitive advantage for CommandBar, with ongoing improvements planned to enhance user experience further.