We introduce auto-evaluator, a simple tool for evaluating question-answering chains built on top of large language models like GPT-3.5-turbo, allowing users to easily assemble and experiment with different chain configurations to optimize QA performance. The tool uses an LLM to score the quality of retrieved documents and answers, providing prompts and results for human inspection and comparison across various tests.