Confident AI introduces "LLM Arena-as-a-Judge," an innovative open-source framework to simplify and enhance the evaluation of large language models (LLMs) by leveraging a pairwise comparison approach. This method, integrated into the DeepEval platform, allows users to efficiently conduct regression testing of LLM applications by selecting the better output rather than relying on complex, single-output evaluation metrics. By employing the Elo rating system and community-based feedback, LLM Arena-as-a-Judge creates a dynamic leaderboard that reflects model preferences. It mitigates biases through randomized positioning and blinded trials, providing a straightforward setup in just ten lines of code. While not a replacement for traditional LLM-as-a-Judge methods, it offers a user-friendly alternative that aligns closely with human judgment, making it particularly suitable for users without specialized knowledge in LLM evaluations.