← All posts

Building a Graph RAG Pipeline with LangChain

A working end-to-end graph RAG pipeline in LangChain — ingest, entity extraction, hybrid retrieval, prompt — plus the parts LangChain quietly doesn't do for you.

By Tulasinath Reddy · February 4, 2026 · Tags: Graph RAG Tutorial LangChain

LangChain has shipped most of the primitives you need for a graph RAG pipeline — LLMGraphTransformer, Neo4jVector, GraphCypherQAChain. It also leaves some sharp edges. This post is a working end-to-end example plus the gotchas that bite teams in the first week.

If you're new to the pattern, read What Is Graph RAG? first.

The stack

  • LangChain for the orchestration glue
  • Neo4j (or FalkorDB, Memgraph) for the graph store
  • pgvector / Chroma for the vector index
  • OpenAI (or any chat model) for entity extraction at ingest and for the answer generation

Step 1: Ingest — chunks + entities, in one pass

from langchain_community.graphs import Neo4jGraph
from langchain_experimental.graph_transformers import LLMGraphTransformer
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_community.vectorstores import Neo4jVector
from langchain.text_splitter import RecursiveCharacterTextSplitter

graph = Neo4jGraph()
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
transformer = LLMGraphTransformer(llm=llm)

docs = load_my_documents()  # whatever loader you use
splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=120)
chunks = splitter.split_documents(docs)

# Extract entities + relationships per chunk
graph_docs = transformer.convert_to_graph_documents(chunks)
graph.add_graph_documents(graph_docs, baseEntityLabel=True, include_source=True)

# Also build the vector index over the same chunks
vstore = Neo4jVector.from_documents(
    chunks, OpenAIEmbeddings(model="text-embedding-3-small"),
    url=NEO4J_URL, username=USER, password=PW,
    index_name="chunk_vector"
)

What this does: every chunk goes into both indexes. LLMGraphTransformer reads each chunk and produces typed nodes (Person, Organization, Concept, ...) plus typed edges. The include_source=True flag links each entity back to the chunk it came from — critical for retrieval later.

Step 2: Retrieve — vector seed + graph traversal

def retrieve(query: str, k: int = 5):
    # 1. Vector retrieval for semantic anchors
    seed_chunks = vstore.similarity_search(query, k=k)

    # 2. Pull entities from those seed chunks
    chunk_ids = [c.metadata['id'] for c in seed_chunks]
    cypher = '''
      MATCH (c:Chunk)-[:MENTIONS]->(e)
      WHERE c.id IN $ids
      WITH DISTINCT e
      MATCH (e)-[r]-(neighbor)<-[:MENTIONS]-(related:Chunk)
      WHERE related.id NOT IN $ids
      RETURN related.text AS text, count(*) AS score
      ORDER BY score DESC LIMIT 5
    '''
    expansion = graph.query(cypher, {'ids': chunk_ids})

    return [c.page_content for c in seed_chunks] + [r['text'] for r in expansion]

The vector search gets the on-topic chunks. The Cypher query expands the candidate set by finding chunks that mention the same entities — that's the multi-hop step that vector-only retrieval misses.

Step 3: Generate — citation-grounded prompt

from langchain_core.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate.from_messages([
    ('system', 'Answer using ONLY the context. For every claim, append [N] where N is the chunk index. If the context does not contain the answer, say "I don't have that information."'),
    ('user', 'Context:\n{context}\n\nQuestion: {question}'),
])

chunks = retrieve(user_question)
context = '\n\n'.join(f'[{i}] {c}' for i, c in enumerate(chunks))
answer = llm.invoke(prompt.format(context=context, question=user_question))

What LangChain doesn't do for you

  • Reranking. Vector + graph expansion gives you a candidate pool. You still need to score and order the combined set. LangChain has ContextualCompressionRetriever but it's slower than a custom reranker on the union.
  • Cost-aware extraction. LLMGraphTransformer calls the LLM once per chunk. On a 100k-chunk corpus that's expensive. Batch, cache by content hash, and prefer gpt-4o-mini over gpt-4o — the entity F1 difference is <3 points.
  • Schema control. By default the transformer invents node and edge types per chunk. Pass allowed_nodes and allowed_relationships to constrain the schema — otherwise your graph fragments into hundreds of one-off types.
  • Hybrid scoring. Combining vector cosine and graph-path scores into a single ranking is your problem. A simple weighted sum (0.7·vector + 0.3·hops) works as a starting point.

The skip-the-plumbing option

Building this is ~400 lines + ongoing operations for two stores. The VertexRAG API wraps the same pattern (chunks + entity-aware graph + hybrid retrieval + reranking) behind one REST call. You bring documents and queries; you don't bring infrastructure. Worth a 10-minute look before you sink a week into wiring this up yourself.

Read more on the VertexRAG blog or try the Graph RAG API.