Human annotations play a crucial role in enhancing AI evaluation pipelines by providing precise feedback, despite their time-consuming and unscalable nature. The process involves collecting data, annotating it using tools like the Phoenix UI or REST API, and using these annotations to build evaluators that check the quality of AI outputs. Through iterative experiments, such as modifying models or prompts, teams can test hypotheses and measure performance changes, leading to improvements in AI systems. Even a small number of annotations can significantly ground evaluations, and tools like Arize-Phoenix facilitate the integration of annotations, evaluators, and experiments into a cohesive workflow. Custom annotation tools enable both technical and non-technical team members to contribute effectively, supporting the transition from inconsistent outputs to reliable, high-quality responses.