Can Text-to-SQL Benchmarks Work on Document Databases? A Couchbase Architecture Case Study

Industry-standard text-to-SQL benchmarks, typically designed for structured databases, face challenges when applied to modern AI-driven query platforms that operate on non-relational, document-oriented data stores like Couchbase. This document outlines the process of adapting the Spider2-Lite benchmark pipeline for Couchbase, a database that uses JSON documents instead of traditional tabular structures. The adaptation involves three key architectural changes: transforming the relational data model into a document-oriented one, reconciling type systems between SQLite and Couchbase's SQL++ query language, and adjusting the query generation process using Couchbase Capella iQ. By preserving the evaluation integrity through result-level comparison rather than direct SQL syntax matching, the adapted benchmark demonstrates that AI query systems can be effectively evaluated on document databases while maintaining the original benchmark's rigor. The case study highlights the benefits of Couchbase's flexible schema and JSON-native querying, emphasizing that modern AI query systems can leverage document-oriented platforms to better align data storage with real-world application structures.