← All posts

Multi-Hop Reasoning: The Real Edge of Graph-Powered RAG

Vector RAG handles 1-hop questions. Multi-hop ones — the interesting kind — require graph traversal. How it works, why vector-only can't reach it, and where it matters most.

By Tulasinath Reddy · May 28, 2026 · Tags: Graph RAG Knowledge Graph Multi-Hop Architecture

Vector RAG is fine when the answer lives in a single chunk. The interesting questions don't. "Which engineer reviewed the PR that broke the deploy in v2.3?" is three hops away from any chunk that contains the full answer — and it's the kind of question every team eventually wants their RAG to handle.

This is the work multi-hop reasoning does. It's the single biggest reason graph-powered RAG exists, and it's the capability that most vector-only systems quietly give up on.

What "multi-hop" actually means

A multi-hop question is one whose answer requires combining facts from multiple documents, where no single document contains the full answer. The number of hops = the number of intermediate joins.

  • 1-hop: "When was the Eiffel Tower built?" → one fact, one document.
  • 2-hop: "What city is the Eiffel Tower in, and what's that city's population?" → location → city → population.
  • 3+ hop: "Which director's films grossed more than the average for sci-fi movies in 2019?" → 2019 sci-fi films → grosses → average → directors above average.

HotpotQA, the standard benchmark, is ~75% 2-hop and ~25% 3+ hop. Real production workloads skew higher — anything that touches relationships (orgs, codebases, customer journeys) is mostly multi-hop.

Why vector RAG can't do it

Vector search ranks chunks by similarity to the query. The intermediate facts that connect the answer to the question often have almost no lexical overlap with the question itself. The query "who reviewed the PR" doesn't look like the chunk "PR #4821 was approved by alice."

So vector retrieval gets you:

  • The chunk mentioning "v2.3 deploy broke" (high similarity)
  • Maybe the chunk mentioning "PR #4821 caused the regression" (medium similarity)
  • Almost never the chunk "alice approved PR #4821" (low similarity to the original question)

The third chunk — the one that contains the actual answer — gets ranked outside top-K because the embedding model has no idea it's the bridge. Increasing top-K to compensate works until you blow your context window or push the real answer out anyway.

How graph traversal fixes it

A knowledge graph stores entities (PRs, engineers, deploys, versions) as typed nodes and their relationships (:authored, :reviewed, :reverted, :fixed) as typed edges. Multi-hop retrieval becomes a literal graph walk:

  1. Anchor: the query mentions "v2.3 deploy broke." Vector search finds the Deploy node for v2.3.
  2. Hop 1: from that Deploy node, follow :caused_by edges → reach the offending PR node (#4821).
  3. Hop 2: from the PR node, follow :reviewed_by edges → reach the engineer node (Alice).
  4. Return: the chunks attached to those three nodes, in order. The LLM now has every fact it needs in ~3 chunks instead of trying to stitch a story from 20.

The retrieval gets more accurate as the question gets harder, which is the opposite of what happens with pure vector search.

What the implementation looks like

At ingest time, every document is parsed twice: once into chunks + embeddings (vector path), once through an entity-extraction pass that produces (subject, predicate, object) triples (graph path). Both go into storage:

document ──┬── chunks → vector index (embedding, chunk_id, doc_id)
           │
           └── triples → graph store (subject, predicate, object, chunk_id)

query ──┬── embedding → top-K chunks   (semantic anchors)
        │
        └── entity-extraction → graph seeds → traversal → chunks N hops away

fuse + rerank → top chunks for the LLM

The two retrievers run in parallel; their candidate sets are unioned, then a reranker scores the combined set using both vector similarity and graph-path relevance. Total added latency vs vector-only: typically 10–30ms for the traversal + rerank pass.

Common objections, honestly answered

"Entity extraction is unreliable."

It was, in 2022. With a modern LLM doing extraction at ingest time (one-shot, structured output), entity F1 on common domains sits at 0.85–0.95. Bad enough for adversarial NER benchmarks, more than good enough for retrieval — you don't need to find every mention, you need to find enough.

"Graphs are hard to maintain."

Hand-curated ontologies are. Extracted graphs aren't — you regenerate them on every ingest. The graph is a derived index, not a primary source of truth.

"Vector search with a bigger top-K does the same thing."

It doesn't, and the failure mode is invisible. With top-K=50 you'll get the right chunk in the retrieved set sometimes, but it'll be ranked at position 38 next to 49 distractors, and the LLM will weight it accordingly. The answer is technically retrievable; the system still gets it wrong.

Where this matters most

  • Enterprise search — "who owns this?" "what depends on this?" "what was changed when?" — almost all multi-hop.
  • Code intelligence — functions, callers, commits, reviewers, deploys form a dense graph; vector-only RAG is famously bad at this.
  • Customer support — joining a customer's plan + integrations + ticket history to answer a single question.
  • Research / literature review — papers cite papers cite papers; the answer to "which findings contradict X?" lives across the citation graph.

How to know if you need it

Audit your last 100 user questions. For each, ask: could a human answer this from a single paragraph? If less than 70% are yes, vector-only retrieval is leaving accuracy on the table. The bigger that gap, the bigger the multi-hop win.

Our free AI Entity Extractor lets you paste any text and visualize the entities + relationships it contains — a quick way to see how dense the underlying graph really is. If a paragraph produces 8+ entities and 5+ relationships, your corpus is graph-shaped and you're losing on retrieval today.

The bottom line

Multi-hop reasoning is the capability vector RAG fundamentally can't reach without an external graph. It's not a niche feature — it's the difference between answering 60% of real questions and 95%. The VertexRAG API builds the graph for you at ingest and runs hybrid retrieval at query time; you write one API call, you get answers that walk the graph automatically.

Read more on the VertexRAG blog or try the Graph RAG API.