ParseBench: The First Document Parsing Benchmark for AI Agents

Document parsing, essential for AI agents interacting with real-world files, often lacks a benchmark that thoroughly evaluates parsing quality across diverse enterprise documents. ParseBench addresses this gap by offering a comprehensive benchmark of approximately 2,000 human-verified pages with over 167,000 test rules across five crucial dimensions: tables, charts, content faithfulness, semantic formatting, and visual grounding. The benchmark compares 14 methods, including vision-language models, specialized document parsers, and LlamaParse, with LlamaParse Agentic performing competitively across all dimensions. It highlights the challenges in accurately extracting data from complex tables and charts, maintaining content faithfulness, preserving meaningful formatting, and ensuring visual grounding for auditability. ParseBench reveals that while content faithfulness is largely addressed, significant issues remain, particularly in chart data extraction and semantic formatting. It also explores the quality-cost tradeoff, noting that LlamaParse offers a cost-effective solution while maintaining high performance. The dataset, evaluation code, and findings are publicly available, encouraging further exploration and improvement in document parsing technologies.