Retrieval-Augmented Generation (RAG) pipelines enhance Large Language Models (LLMs) by integrating proprietary data, transforming them into specialized internal tools for enterprises. However, this introduces significant security challenges, particularly the risk of unauthorized data access due to intricate access control policies. Traditional pre-filtering techniques, which attempt to incorporate permissions into vector metadata, struggle to scale due to issues like synchronization lag and metadata complexity. Instead, a more effective approach is post-retrieval filtering, which separates authorization from the retrieval process. This involves first retrieving data based on semantic similarity and then applying a real-time authorization check using a Relationship-Based Access Control (ReBAC) system, inspired by Google's Zanzibar, to ensure only authorized data is accessible. This method balances speed and security, enabling enterprises to utilize RAG pipelines without compromising performance or data protection.