ExtractBench Pro: The Long Tail of Document Extraction

ExtractBench Pro is a new 100-document extension of the ExtractBench benchmark designed to test long-tail document extraction challenges that were underrepresented in the original dataset, including cross-page rules, organization-specific terminology, derived calculations, diagram-based relationships, complex charts, and unusually long chart sheets. Its documents are synthetically generated through deterministic pipelines that combine anonymized real-world formats and styles with configurable content, enabling reliable ground-truth annotations for both extracted values and their source locations. Testing of twelve systems, including extraction APIs, coding agents, and three LlamaExtract tiers, found that performance fell substantially compared with the original benchmark, illustrating the difficulty of these cases; coding agents achieved the strongest overall extraction and grounding scores but generally at higher cost and sometimes beyond the 30-minute limit, while LlamaExtract occupied the reported low-cost performance frontier. The release also introduces the percentage of documents with perfect Value F1 as a metric, reflecting the finding that systems often either solve a difficult document nearly completely or fail on many relevant fields, and the dataset, code, and paper are available publicly for further evaluation.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.