Loop is an AI assistant integrated into Braintrust, designed to enhance evaluation workflows by providing insights from production data, optimizing prompts, and generating custom charts while maintaining accessibility for both technical and non-technical users. Despite promising early metrics, a low acceptance rate for the "Optimize this prompt" feature prompted Mengying from the growth team and design engineer David to investigate user interactions. Through manual reviews, they identified that Loop's default behavior of optimizing all prompts when multiple were present led to user dissatisfaction. To address this, they developed a scoring framework to evaluate user interactions, identified patterns of low satisfaction, and created a targeted dataset for testing solutions. By refining Loop's behavior iteratively and using prompt versioning, they improved user satisfaction, as evidenced by an increase in average conversation scores from 2.1 to 4.3 and reduced computational costs. Their findings highlighted the importance of combining technical and non-technical perspectives in resolving AI product issues and demonstrated how Loop's features can be leveraged to optimize AI applications efficiently.