To reliably evaluate the quality of a question answering (QA) system, it's essential to use quantifiable metrics coupled with a labeled evaluation dataset. This allows for informed assessments of the system's quality, comparison of different models, and identification of underperforming components. The QA system consists of a retriever and reader model chained together in a pipeline object, where the retriever chooses documents from a database based on a query, and the reader extracts the correct answer from those documents. Evaluation datasets should be manually annotated with correct answers, and there are two evaluation modes: open domain (single document QA) and closed domain (multiple document QA). Metrics such as recall, mean reciprocal rank, exact match, F1 score, accuracy, and semantic answer similarity can be used to evaluate the retriever and reader models individually or in combination. These metrics provide insight into the system's performance, helping developers optimize their pipeline for better results.