Developing Copilot: What AI Engineers Can Learn from Our Experience Building An AI Assistant

Developing an AI assistant tailored for data scientists and AI engineers called Arize Copilot involved numerous challenges and valuable lessons about developing with LLMs. The tool is designed to assist users in troubleshooting and improving their models and applications through an agentic workflow, leveraging the Completions API from OpenAI for better control over state management. Lessons learned include managing state effectively, handling model swaps cautiously, using prompt templates with clear instructions and guidelines, incorporating data into prompts in a structured format, configuring function calls explicitly, implementing streaming efficiently, focusing on user experience, and utilizing testing strategies with datasets and automated workflows.