Benchmarking Question/Answering Over CSV Data

A deep dive into question-answering over tabular data using CSVs outlines the challenges and solutions involved in creating a natural language interface for such data. Initially, a Streamlit app was developed to gather real questions from users, revealing issues like unclear question types and evaluation difficulties due to a lack of data and metrics. The authors used LangSmith to construct a dataset and evaluate solutions, employing LLMs for correctness assessment. The final solution was a custom agent using OpenAI functions, a Python REPL, and a retriever, which allowed for both text and numerical data queries. Despite some initial challenges, the improved system demonstrated effective performance, particularly in handling complex queries about the Titanic dataset, though it occasionally required dataset-specific prompts for optimal results. The entire project, including the app, dataset, and evaluation script, was open-sourced to aid further development in this area.