Classifying User Intent with Categorical LLM-as-a-Judge

Categorical LLM-as-a-Judge scores have been introduced as a feature to classify user intent by returning category labels instead of just numeric scores, enhancing the ability to filter, analyze, and visualize user interactions in support applications. This functionality has been implemented in a Langfuse support chatbot demo, categorizing user queries into six distinct types: conceptual questions, implementation questions, self-hosting inquiries, pricing and comparison questions, UI feedback, and irrelevant content. The guide explains how to configure the evaluator by defining category values and setting up dashboards to track and analyze user intents, enabling more targeted user support and feedback management. The evaluator is designed to classify each user message into one primary category, with rules to resolve ambiguities and ensure actionable insights, and is available across all Langfuse plans, supported by comprehensive documentation and a public demo project.