October 2021 Summaries

This new metric for evaluating question answering systems is called Semantic Answer Similarity (SAS). SAS measures the semantic similarity between two answer strings, rather than just their lexical overlap. This makes it a better approximation of human judgment than existing metrics like Exact Match (EM) and F1. SAS uses a cross-encoder architecture that leverages a pre-trained semantic text similarity model to assess the similarity of two strings. The SAS metric returns a score between zero and one, with higher scores indicating greater semantic similarity. To use SAS in Haystack, users can initialize the SAS model together with the EvalAnswers() node and run the pipeline to evaluate their question answering system. While SAS has its strengths, it also has limitations, such as potentially awarding high scores to strings that are semantically similar but not accurate. Nevertheless, SAS can provide a better understanding of how well a question answering system is doing compared to existing metrics like F1 and EM.

The latest advancements in conversational AI have led to a surge in chatbot adoption, driven by tools like Rasa, the leading conversational AI platform. Building a chatbot that can identify information-seeking intents and perform question answering on a large-scale database of documents is now possible with the integration of Haystack, a powerful question-answering technology. By combining Rasa's dialogue flow management capabilities with Haystack's ability to extract relevant information from unstructured text, developers can create more robust chatbots capable of handling long-tail informational user requests. This integration enables chatbots to trigger external services like Haystack, providing context-dependent responses and improving overall conversational flow. The process involves setting up a Haystack service accessible via REST API, configuring Rasa to call the Haystack API, defining intents and actions, and training the model to recognize information-seeking utterances. With this powerful combination, developers can build more intelligent chatbots that provide accurate and informative responses to users' queries.

To properly index documents for large-scale question answering systems with Haystack, it's essential to learn how to clean, split, and index text data. Haystack provides tools like PreProcessor, Crawler, Converter, and DocumentStore to make working with text data easier. The document indexing process can be achieved through different methods: defining an indexing pipeline via YAML configuration, using the REST API for continuous indexing of documents, or directly adding documents to the database. Each method has its own advantages and is suited for specific use cases. By choosing the right approach, developers can successfully deploy a Haystack question answering system with their own dataset.

At a tech company called deepset, remote work is not just accepted but also actively encouraged as part of their culture. The company has been fully remote since its inception and values individual focus and creativity while fostering interaction and collaboration throughout the day. To facilitate this, they have established three core principles: Core Hours Policy, Periodic Exchange of Ideas and Feedback Sessions, and Quarterly In-person Team Building and Strategy Sessions. These principles aim to address the challenges of remote work, such as finding new ways to hold creative whiteboard sessions and team alignment meetings. The company also organizes regular virtual lunches and holds daily standup calls with a focus on social interaction and collaboration. Their quarterly in-person event, Re:base, provides an opportunity for high-level strategy discussions, team building, and bonding among team members. Overall, deepset's remote work culture is designed to promote work-life balance, autonomy, and flexibility while also fostering a sense of community and trust among team members.