July 2024 Summaries

The Sanlam Studios team has developed an AI financial coach that educates customers, generates leads, and offers empathetic judgment-free financial support. The agent uses a hybrid UX/UI interface to engage users without leaving the coaching flow, making it more engaging for customers. It also generates personalized offers based on user data and offers proactive follow-ups with human advisors. The team built the agent using Voiceflow, leveraging LLMs, prompt chaining, and contextual summarization techniques. They experimented with safe bets, threw away initial agents, and worked closely with legal and compliance teams to ensure defensible choices. The AI coach has seen significant success, with 45% of offers leading to conversations with human advisors, zero follow-up questions from users, and double the conversation engagement compared to most agents. The team plans to expand the agent into new verticals and channels, including WhatsApp, and continue to innovate with AI.

Voiceflow has partnered with Anthropic to integrate their Claude model family into its collaborative conversational AI platform. This integration enables enterprise teams to quickly build and deploy generative AI applications for various use cases, such as customer support automation and internal task automation, using the full access to the Claude model family. With this partnership, Voiceflow customers will gain joint model and build expertise, use case-specific guidance, and control over costs and performance. The integration is expected to help teams automate complex interactions and unlock the value of AI automation in the enterprise.

Researchers explored ways to improve the accuracy of Large Language Model (LLM) intent classification prompts. They experimented with various modifications to descriptions, including adding prefixes, suffixes, and capitalization, as well as using AI-generated descriptions. The results showed that these modifications can lead to small but measurable improvements in accuracy, particularly when combined with other techniques such as structuring formatting. The top-performing configurations involved adding prefixes, using AI-generated descriptions, and paying attention to edge cases. While the gains were not dramatic, the study suggests that optimizing prompt design is an important area of research for improving LLM performance.

AI hype took off in 2023 with ChatGPT's rapid user growth, leading executives to demand proof of concepts and investment. This surge created pressure on teams to develop AI models, prompting concerns about ROI, integrations with legacy systems, and workflow impacts. To treat AI agents as core products, organizations must prioritize collaboration, align with customer needs, define success metrics, and mitigate risks. Building a dedicated team with the right skills is crucial for creating effective AI solutions that deliver value to organizational goals. Denys emphasizes the importance of mapping the AI product journey from business case to production, de-risking agents through checkpoints like user acceptance testing and private beta reviews, and measuring agent performance using metrics such as time-to-response and resolution rate/speed. By treating AI agents as core products and applying common product management principles, organizations can create useful and effective solutions that drive value and ROI.

To effectively scale an AI agent, teams should start small with a minimal viable product (MVP) and iteratively add more use cases as they master each one. This approach allows for cost-effective scaling and reduces the risk of over-investing in complex solutions. By organizing their team around code-focused senior product specialists who analyze performance data and refine processes, teams like Trilogy have successfully automated 60% of customer support in under two months. When choosing LLM models, consider using RAG to provide context, selecting versions that work best for specific tasks, testing different models for various use cases, and balancing prompt engineering costs against model upgrades. GPU hardware can be essential for large-scale AI projects, but smaller GPUs or serverless approaches can suffice for many use cases. A budget should prioritize time, effort, and strategic thinking over risk aversion, with allocations for keeping up with the AI landscape, evaluation-driven development, experimentation, and known problems. By starting small, scaling up carefully, and prioritizing continuous learning and experimentation, teams can achieve impressive results with limited budgets.