September 2021 Summaries

To reliably evaluate the quality of a question answering (QA) system, it's essential to use quantifiable metrics coupled with a labeled evaluation dataset. This allows for informed assessments of the system's quality, comparison of different models, and identification of underperforming components. The QA system consists of a retriever and reader model chained together in a pipeline object, where the retriever chooses documents from a database based on a query, and the reader extracts the correct answer from those documents. Evaluation datasets should be manually annotated with correct answers, and there are two evaluation modes: open domain (single document QA) and closed domain (multiple document QA). Metrics such as recall, mean reciprocal rank, exact match, F1 score, accuracy, and semantic answer similarity can be used to evaluate the retriever and reader models individually or in combination. These metrics provide insight into the system's performance, helping developers optimize their pipeline for better results.

Haystack provides a free annotation tool to assist in creating high-quality question answering (QA) datasets, making the process quicker and easier. Data labeling is necessary for machine learning models, involving the identification of raw data and assigning labels so that the model can properly interpret the context. The Haystack annotation tool helps coordinate team work by setting up standard questions and assigning members sets of documents. It allows users to create unique or standard questions, mark answer spans, and export the annotated dataset in SQuAD format. A well-designed question and answer pair consists of a fact-seeking question aiming to fill a gap in knowledge and an answer that is shorter rather than longer. The tool enables users to build a tailored QA pipeline using their own datasets.

Haystack, a question answering system, can be extended with various nodes to leverage pre-trained Transformer-based language models. The Classification node can classify answer candidates and add metadata labels such as sentiment analysis or named entity recognition. The Summarizer node uses a pre-trained model to create summaries of documents returned by the Retriever. The Generator node allows creating answers from scratch, working in tandem with dense passage retrieval (DPR) method. These nodes can be combined to create a more efficient and accurate question answering system, enabling features such as sentiment analysis, summarization, natural language generation, and multilingual support.

Configuring Haystack pipelines through YAML makes it easier to set up a question answering system, allowing for quick tweaks, parameter management in production, and distributed execution. YAML is particularly useful for conducting experiments as it allows for easy adjustment of pipeline components without modifying source code. By defining components such as Readers and Retrievers, and pipelines that chain them together, users can create complex neural search systems using a configuration file. The use of YAML also enables the deployment of Haystack pipelines with a REST API, allowing for easy import, modification, and export of configurations between experimentation and production environments.

Leveraging transformer models can help users find answers in FAQs by understanding the intent behind queries. Traditional keyword-based search systems are limited, as they require exact word matches, which can be challenging for users who use different wording to ask questions. Transformer-based language models like BERT and RoBERTa can go beyond literal matching and grasp the meaning behind a query, making them suitable for question answering tasks. A semantic FAQ search system uses dense retrieval models, such as Sentence-transformers, to represent FAQs as vectors in a high-dimensional embedding space, allowing the system to compare queries to questions and return answer associated with that question. An example implementation using Haystack demonstrates how to build an FAQ pipeline on top of a small dataset of FAQs about chocolate, showcasing how the system can match queries to relevant answers despite minor differences in wording.