Why RAG Failures Look Like Model Failures 

By Brad Bell, Principal Observability Lead

The incident channel lights up on a Tuesday. The AI assistant told a customer that a discontinued product ships in two weeks. The on-call engineer pulls the conversation, winces, and posts the diagnosis everyone expects:  the model hallucinated. 

So the team does what teams do. They tighten the system prompt. They bump to the newer model version. Someone proposes switching providers, and someone else builds a comparison spreadsheet. Two weeks of work ships, everyone feels better, and on a Thursday three weeks later the assistant confidently quotes a price from last year’s catalog. 

Here is the uncomfortable pattern we keep finding when we trace these incidents:  the model was never the problem. The retrieval pipeline fetched a stale document, or a low-relevance one, or the right document from the wrong year, and the model did exactly what it is built to do. It answered fluently and correctly given the context it was handed. Garbage in, fluent garbage out. This is not a stray anecdote: industry analysis and customer incident reviews consistently point to retrieval, not the language model, as the dominant source of enterprise RAG failures, and Stanford research has shown that even specialized, retrieval-grounded legal AI tools still hallucinate in 17 to 33% of cases. The model is rarely the part that broke. 

The misdiagnosis is structural, not careless 

Smart teams make this mistake for a structural reason: they can only debug what they can see, and what they can see is the model. Token counts, latency, the system prompt, the final answer. All of it logged, all of it inspectable, all of it pointing at the most heavily instrumented component in the stack. 

Meanwhile the retrieval pipeline (embed the query, search the vector store, rerank the candidates, assemble the context window) runs in the dark.  Most teams cannot answer the most basic forensic question about a bad answer: what documents did retrieval return for that request, and were they any good? 

When the symptom is visible in one place and the cause lives in another, every investigation drifts toward the symptom. That is not a discipline problem. It is an instrumentation problem, and it has an instrumentation fix. 

What actually breaks upstream 

The retrieval failure modes are mundane, which is exactly why they hide so well. In rough order of how often we find them: 

  • Stale documents. The source content changed; the index did not. The assistant quotes the old refund policy with total confidence because, as far as its context is concerned, that is the current policy. 
  • Low-relevance retrieval. The query embedded poorly or the corpus is thin in that area, so the top results are tangentially related at best. The model stretches to be helpful with weak material, which reads as hallucination and is closer to improvisation. 
  • Right document, wrong slice. Chunking split a table from its header or a caveat from its rule. The retrieved fragment is technically from the right source and substantively misleading. 
  • Version collisions. Three versions of the pricing sheet live in the corpus and retrieval has no recency or authority signal to choose between them. Some days it picks the right one. 
  • Silent pipeline drift. An embedding model changed, an ingestion job half-failed, a permissions filter started excluding the most useful folder.  Nothing alerted, because nothing was watching. 

Notice what is absent from that list: the model. It can certainly fail on its own, and genuine hallucination is real. But when we work backward from production quality incidents in RAG systems, the trail leads upstream far more often than the postmortems predicted. The model is where the failure becomes visible, not where it happens. 

Why the fixes never stick 

This is also why the prompt-and-model-shuffle response feels productive and changes nothing. A better system prompt cannot rescue an answer grounded in a stale document. A newer model summarizes the wrong context more eloquently. Switching providers re-rolls the dice on a game whose deck was stacked upstream. 

Worse, sometimes the shuffle appears to work. The index happened to refresh that week, or traffic shifted away from the broken corpus area, and the fix gets credit for a coincidence. Now the team’s mental model is wrong twice: they believe the model was the problem, and they believe they solved it. The next incident starts the loop again with extra confidence. 

The test you can run this afternoon 

There is a cheap, fast way to find out which side of this pattern your system is on, and it requires no new tooling to attempt: 

Pull your last five bad answers. The ones users flagged, the ones that made it to the incident channel, the embarrassing ones in the screenshot folder. For each, answer one question: what did retrieval return for that request? 

Three outcomes, and every one of them is useful. If you can check and retrieval was clean, you have a genuine model or prompt problem, now proven instead of assumed, and the model-side tools are the right ones. If you can check and retrieval returned garbage, you just located your actual failure domain, and you can stop paying the prompt-engineering tax on an infrastructure problem. And if you cannot check at all, that is the finding:  the most failure-prone component in your AI stack is also the only one you cannot see. In our experience, most teams that run this test land in the third outcome. 

What retrieval telemetry actually looks like 

Closing the gap means treating retrieval as the production system it is. The signals worth capturing per request are not exotic: 

  • The query as embedded, not just as the user typed it. 
  • The documents returned:  identifiers, sources, and the content or a content hash, captured under whatever compliance posture your data requires. 
  • Relevance scores from the retriever and the reranker, so “how good were the candidates” is a number, not a vibe. 
  • Document freshness:  when the source last changed and when it was last indexed, which together make staleness alertable. 
  • Trace context linking the retrieval span to the conversation and the model call, so one bad answer unrolls into its full causal chain. 

The encouraging part: this no longer requires inventing anything. The OpenTelemetry gen_ai semantic conventions give retrieval spans a standard shape,the major frameworks emit much of it with configuration rather than code, and the telemetry flows through the same pipelines as your existing traces. Instrumented this way, retrieval quality stops being a forensic archaeology project and becomes a dashboard, then a trend, then an alert that fires before a customer does. 

It also future-proofs the work. Retrieval telemetry on open conventions is exactly the data that automated evaluation and AI-assisted operations will consume as those capabilities mature. Instrument it once for debugging today and you have already built the input layer for the automation you will want in eighteen months. 

The reframe that changes the postmortem 

None of this argues that models are innocent or that evaluation does not matter. It argues for a diagnostic order. When the answer is wrong, check what the model was given before debating what the model did. Teams that adopt that one habit stop burning sprints on prompt changes that were never going to hold, and their postmortems start naming root causes that stay fixed. 

If you want to know where your own system stands, start with the five bad answers test;  it costs an afternoon. And if the honest answer is that you cannot see your retrieval pipeline at all, that is worth knowing before the next incident writes it up for you. We built a three-minute self-assessment that scores retrieval visibility alongside the four other signals that decide whether your AI is observable at all;  it is a fair place to start the conversation, with us or just with your own team. 

About the Author

Brad Bell is the Principal Observability Lead at TekStream Solutions, a Digital Resilience partner that helps organizations operate, recover, and adapt with confidence by modernizing, securing, and optimizing their digital environments. His 29 years of experience spans large-scale telecommunications infrastructure, enterprise cloud strategy, and operational transformation across industries ranging from financial services to quick-service restaurants — work that included scaling monitoring platforms to 12,000+ nodes, architecting next-generation networks, and turning reactive ops teams into reliability engineering organizations. A Grafana Certified Solutions Architect and PreSales Solution Architect, he teaches SRE immersion courses at major financial institutions and consults on observability strategy at the enterprise level, helping clients navigate the convergence of cost pressure, tool sprawl, and AI-driven operations that is reshaping how enterprises think about reliability. His perspective isn’t theoretical — it comes from having been the person on the other end of a 2 AM page, and from watching organizations make expensive, avoidable mistakes with their observability investments. At TekStream, his focus is helping enterprises build platforms that solve today’s problems and hold up against where the industry is headed.