Building Topic Discovery into the Hex Context Agent

Hex has developed a sophisticated topic discovery system to enhance its Context Agent, allowing users to ask questions in natural language and receive accurate responses. Initially, the team attempted a machine learning approach with traditional clustering algorithms like HDBSCAN and UMAP, but these proved inefficient due to high computational demands. Instead, they pivoted to a simpler, more effective method using large language models (LLMs) that improved topic quality by separating topic discovery and thread classification into distinct stages. This system operates by analyzing thread summaries to propose topics and then classifying new threads accordingly, ensuring ongoing topic refinement. Despite the limitations of a flat topic structure, customer feedback has been positive, and Hex plans to explore further enhancements, including topic-specific guides and evaluation suites, to better understand and address user needs. This approach underscores the value of leveraging LLMs to simplify complex processes, reducing the gap between concept and implementation, and reflects Hex's commitment to creating impactful data products.