Evaluating the functional performance of Large Language Models (LLMs) is crucial in ensuring they continue to work well over time, amid changing trends in production environments. However, producing effective metrics for evaluating LLMs poses significant challenges due to the difficulty in obtaining a stable ground truth and tailoring evaluations to specific use cases. To address this, various evaluation approaches can be considered, including code-based, LLM-as-a-judge, and human-in-the-loop methods. These approaches help characterize the application's performance across different dimensions such as accuracy, relevancy, coherence, toxicity, and sentiment in inputs and outputs. Specifically, context-specific evaluations assess the model's ability to retrieve relevant context and infer from it appropriately, while needle-in-the-haystack tests evaluate the model's retrieval capabilities, and faithfulness evaluations test the model's self-consistency within an LLM-as-a-judge framework. User experience evaluations leverage user feedback data to measure the effectiveness of responses, topic relevancy evaluates the relevance of questions or answers to the application's established domain, and security and safety evaluations monitor for breaches and toxicity in inputs and outputs. By creating a comprehensive monitoring framework, teams can obtain continuous visibility into their LLM application's functional performance and optimize its parameters to improve accuracy, coherence, and user experience.