Debugging Our Docs RAG, Part 2: Testing New Generation Models

In this analysis, a focused evaluation was conducted on the generative models used in a Retrieval-Augmented Generation (RAG) system, revealing significant improvements when newer models like Gemini 3 and GPT-5.2 were employed, compared to a legacy model from 2023. Despite achieving a performance increase from 3 to 10 out of 14 correct answers without altering the retrieval pipeline, persistent failure modes such as "needle-in-a-haystack" retrieval failures, hallucinations in multiple-choice scenarios, and omissions of critical details remained. The study underscored that while upgrading generative models offers substantial gains in handling noisy contexts and simple queries, there is a performance ceiling that suggests further improvements will require enhancements in the retrieval process, specifically through better embedding models to address nuanced documentation differentiation. The findings indicate that while model upgrades are beneficial, they are insufficient alone, pointing to the need for more comprehensive iterations that include retrieval assessment.