Authorization for RAG at Scale: Why You Shouldn't Sync Every Document

RAG (Retrieval-Augmented Generation) systems face challenges in scaling authorization for document access due to high-cardinality resources resulting from breaking down documents into numerous components. Traditional fine-grained authorization (FGA) systems, which rely on relationship graphs, become bottlenecks because they require syncing every document with an external authorization system, leading to operational fragility. To address these issues, a scalable approach combines hierarchical authorization with local metadata filtering, whereby access is granted at the level of collections, workspaces, or folders, and documents inherit this access. This method involves registering stable parent resources and assigning roles at this level, while filtering is conducted locally by the vector database optimized for high-cardinality operations. This approach maintains a small and stable authorization graph while enabling the vector store to handle scaling. For applications that require document-level permissions, specific documents can be registered as resources under this framework, with document-level authorization as the exception rather than the norm. The strategy allows for starting with simple role-based access control and integrating resource hierarchies as the application grows, avoiding the need for significant rewrites or migrations.