June 2024 Summaries

In a recent episode of the High Agency Podcast, host Raza Habib conversed with Shawn Wang, also known as Swyx, discussing the emerging role of AI Engineers, a distinct position that bridges traditional machine learning (ML) engineering and product-focused development. Swyx, a prominent figure in AI engineering, emphasized that while AI Engineers might face skepticism, their focus on integrating AI capabilities into products is crucial in today's landscape. Unlike ML Engineers who focus on model development, AI Engineers work with pre-trained models and APIs to solve specific product challenges, often requiring rapid prototyping and iteration. Swyx suggested a team composition of four AI Engineers to one ML Engineer to balance model depth with product application. He highlighted several industry trends, such as the commodification of AI capabilities, improvements in inference speeds, expanded context windows, and the significance of multimodal AI. Additionally, Swyx introduced "temperature 2" use cases where AI's creative outputs are leveraged as features, encouraging innovation in AI applications. The conversation also covered the importance of moving quickly to deploy AI products, the evolving composition of AI product teams, and the potential impact of vertical AI startups compared to horizontal ones. The discussion culminated in anticipation of the upcoming AI Engineer World Fair Summit, which aims to explore these topics further and foster community engagement within the AI engineering field.

In evaluating AI products built with large language models (LLMs), Hex AI, led by Bryan Bischof, has developed a unique approach that focuses on breaking down the evaluation process into granular, user-centric components rather than relying on a single "god metric." This methodology ensures a comprehensive assessment of AI agents, which are designed to automate complex data analysis tasks by generating SQL queries and creating visualizations. The success of Hex's AI agents stems from a strategic system design that includes mapping tools to existing workflows, creating reactive directed acyclic graphs (DAGs) to track task dependencies, and keeping humans in the loop to correct AI actions. Instead of simplifying evaluations into one metric, Hex employs a suite of binary evaluators that align with the ideal user experience, ensuring the AI product delivers true value. Bischof emphasizes the importance of immersing oneself in data to uncover insights, advocating for regular team engagement with evaluation data to improve AI product performance. This approach, supported by platforms like Humanloop, which facilitates logging and observability, demonstrates that a thoughtful, data-driven evaluation can lead to reliable AI agent deployment.

Ironclad, a pioneer in deploying generative AI in legal tech, has achieved significant success by integrating AI tools into familiar interfaces like Microsoft Word, making it easier for lawyers to adopt their technology. Founded by Raza Habib and Cai Gogwilt, Ironclad offers AI products that automate contract review and negotiation, with over 50% of contracts at top clients, including OpenAI, being processed by AI. Despite initial challenges that nearly led to abandoning their AI agents project, Ironclad succeeded by developing Rivet, an open-source AI agent builder that simplifies debugging and iteration. Gogwilt advises product leaders to move quickly and boldly in adopting AI, emphasizing the importance of fitting AI features into existing workflows and launching early to gather user feedback. The company's experience underscores the necessity of robust tooling for evaluating and debugging AI applications to ensure reliability and alignment with user needs.