We Spent a Decade Making AI Feel Instant. Here's What We Learned.

Sri Raghu Malireddi and Harsha Nalluru, former leads at Grammarly and Microsoft respectively, developed Moss to address the latency issues faced by AI agents in delivering real-time interactions. Traditional reliance on network-based retrieval from vector databases, such as Pinecone and Weaviate, resulted in delays that disrupted user experiences in chatbots, voice agents, and copilots. By embedding the semantic search index within the same process as the AI agent, Moss eliminates the need for network hops, achieving sub-10ms retrieval times. Built with Rust and WebAssembly for performance and portability, Moss provides a compact and efficient solution for instant local lookups, enhancing the responsiveness of AI systems. Launched through Y Combinator, Moss is gaining traction with platforms where retrieval latency critically impacts user experience, promising further insights into AI architecture in their upcoming series.