The Hallucination Problem in RAG — and How to Fix It

RAG hallucinations are mostly a retrieval problem, not an LLM problem. The three root causes, how to detect them, and four fixes that actually work in production.

By Tulasinath Reddy · January 14, 2026 · Tags: Graph RAG Architecture Hallucinations

Teams reach for bigger models when their RAG hallucinates. They should be looking at their retriever. In our internal audits of broken RAG pipelines, ~80% of confident-but-wrong answers trace back to a retrieval failure — the LLM was given the wrong context and dutifully invented an answer to fit.

This post is the diagnostic + the fix. Three root causes, how to detect each one, and four production-grade interventions that move the needle.

What hallucination actually is in RAG

An LLM hallucinates when its output asserts a fact that isn't supported by its inputs. In a RAG system, the inputs are the retrieved chunks plus the system prompt. So a RAG hallucination is, definitionally, a statement the model made up from its parametric memory because the retrieved chunks didn't say it.

This framing matters: "the model is bad" is rarely the right hypothesis. "The model got bad context" usually is.

The three root causes

1. The right chunk wasn't retrieved (missing context)

The most common failure. The answer exists in your corpus but the retriever ranked the relevant chunk outside top-K. The LLM, having no on-topic context, freestyles from training data.

Detect: calculate recall@K against a labeled question set — for each question, was the ground-truth chunk in top-K? Anything under 0.8 is a recall problem dressed up as a hallucination problem.

2. Conflicting chunks were retrieved

Two chunks contradict each other (a doc and its updated version; conflicting policies; multiple sources with slightly different numbers). The LLM picks one, blends them, or invents a reconciliation. All three are wrong.

Detect: faithfulness scoring catches some of these — when the model's answer isn't supported by any single retrieved chunk, it's likely a blend.

3. No relevant chunks exist (out-of-corpus query)

The user asks something your corpus doesn't cover. Vector search will always return top-K results — there's no built-in "no good matches" signal. The LLM gets garbage context and answers anyway.

Detect: look at the similarity score of the top retrieved chunk. If it's below your typical floor (e.g. cosine < 0.7), the query is likely out-of-corpus.

Four fixes that work

Fix 1: Improve recall first

Before anything else, get recall@10 above 0.9 on a representative eval set. Hybrid retrieval (vector + BM25), better chunking, or — for relationship-heavy corpora — adding a knowledge graph layer (graph RAG) does more for hallucination rate than any prompt engineering.

Fix 2: Citation-grounded prompts

Require the model to cite the chunk ID for every claim. "For each statement in your answer, append [chunk_N] where N is the source." This both constrains the model (it can't claim what it can't cite) and gives you a deterministic faithfulness signal at output time.

Fix 3: An "I don't know" affordance

Give the model explicit permission to refuse. "If the retrieved context doesn't contain the answer, respond exactly: 'I don't have that information.'" Combined with a similarity-floor check on the retrieved chunks, this cleanly handles out-of-corpus queries.

Fix 4: Two-pass verification

Generate the answer, then run a separate LLM call that scores each claim against the context. Reject (and retry, or fall back) if the faithfulness score is below your threshold. Doubles inference cost but reduces hallucination rate by 60–80% in our experience. See the eval guide for the scoring pattern.

Why graph RAG cuts hallucinations specifically

Hallucinations on multi-hop questions are usually composition errors — the model joins two facts that look like they should connect but don't. Graph RAG removes the inference: the retriever surfaces the actual edge between the two entities, so the model isn't asked to guess at the join. Less inference room = fewer hallucinations on exactly the questions where vector RAG fails worst.

What not to do

Don't just raise top-K. More chunks = more noise. Above K=15 the model gets worse, not better.
Don't fine-tune to fix recall. You're paying training cost to compensate for an indexing bug.
Don't prompt-engineer your way around retrieval. "Be more careful" is not a substitute for getting the right chunk in the context.

Check your own content

Our free RAG Evaluation tool gives you a 0–100 score on how retrieval-friendly any piece of text is. Low scores on entity density and structural quality are leading indicators that your corpus will trigger the failure modes above. The VertexRAG API ships citation-grounded responses out of the box — every answer comes back with the chunks that supported it.

Read more on the VertexRAG blog or try the Graph RAG API.