How We Built an Agent Skill to Synthesize what Langfuse Users want

Langfuse developed an AI agent skill to consolidate and synthesize user feedback from various sources, such as GitHub issues, support tickets, and meeting notes, into a weekly digest. The AI skill operates by fetching data through scripts and processing it according to instructions in the SKILL.md file, effectively aggregating similar issues into coherent topics, and linking each item back to its original source for context. Initially, the output was vague and categorized by product area, but through iterative improvements using feedback and Langfuse's monitoring, the digest evolved to present clearer, actionable insights grouped by decision type. The evaluation of the digest's effectiveness focused on the actionability, source citation, and consistency in the number of categories, revealing challenges such as incorrect links, misinterpretation of recent activity, and implicit assumptions in category names. By addressing these issues, the team was able to refine the AI skill and enhance the reliability of the digest, ensuring it provided valuable insights for engineers and stakeholders.