How to build agentic AI when your data can’t leave the network

Small language models (SLMs) offer a viable solution for organizations needing AI capabilities while adhering to privacy and data locality constraints. Unlike large language models (LLMs), which often require cloud access and can pose privacy risks, SLMs can be deployed locally or on modest on-premise servers, enabling private and secure decision-making processes. Research highlights that SLMs, particularly those in the 1-3 billion parameter range, excel in tasks requiring reasoning, classification, and retrieval, with performance not solely determined by size but by training strategies and inference techniques. This allows for the creation of agentic systems where reasoning, retrieval, and expression are handled separately by specialized models, ensuring that sensitive data remains within the network while offering cost efficiency and scalability. The architecture proposed uses SLMs for local processing, only resorting to cloud-based LLMs for optional expressivity, thus providing an effective AI solution for enterprises with stringent data policies.