Fin fixed the fake refund promises without losing the upside

Fin, an AI support agent developed by Intercom, has improved its capabilities through a rigorous experimentation framework led by Principal Machine Learning Scientist Pedro Tabacof. The AI agent, which now generates over $100 million in annual recurring revenue and serves more than 10,000 customers, emphasizes the importance of large-scale A/B testing to refine its functionalities, given the non-deterministic nature of AI systems where traditional unit tests fall short. Through continuous testing, even for minor changes, Fin has discovered counterintuitive insights, such as the benefits of increased latency, which can enhance user perception by making interactions appear more thoughtful and human-like. A notable experiment involved increasing the conversation history context to improve answer quality, initially leading to unintended hallucinations like false refund promises. By revisiting and refining prompts, the team managed to retain the positive improvements without the downsides, demonstrating that failed experiments often have a path to success through targeted adjustments. Fin's culture of experimentation, viewed as essential for quality rather than a hindrance to speed, allows for innovation and learning from failures, underpinned by leadership that values data-driven decision-making.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.