← All posts

RAG Chunking Strategies That Actually Work

Chunking quietly decides whether your RAG works. Four strategies that perform in production, when each is right, and the mistake everyone makes once.

By Tulasinath Reddy · December 22, 2025 · Tags: Tutorial Architecture Chunking

Chunking is the unsexy step that quietly decides whether your RAG works or doesn't. Get it wrong and even the best embedding model retrieves the right document but the wrong paragraph; the LLM then either misses the answer or hallucinates around it. Get it right and ~half of your retrieval accuracy problems disappear before you tune anything else.

This is the short, practical guide. Four strategies that actually work in production, when each one is right, and the mistake every team makes once.

Why chunking matters more than people think

The LLM never sees your documents — it sees chunks. The chunk size determines what the embedding represents, what fits in the context window, and how much surrounding context the model has when generating. Three things go wrong with bad chunking:

  • Splits across the answer. The sentence that answers the question lives half in chunk N and half in chunk N+1, so neither one scores well for retrieval.
  • Buries the answer in noise. An oversized chunk dilutes the embedding — the relevant sentence is one of 50, and similarity drops.
  • Loses the surrounding context. An undersized chunk retrieves the right fact but strips away the qualifying "...except in California" that made it true.

Strategy 1: Fixed-size chunking

Cut every document into chunks of N tokens (typically 256–1024) with M-token overlap (typically 10–20% of N). Simplest possible approach; it's what every "hello world" RAG tutorial uses.

Use it when: you're prototyping, your corpus is homogeneous prose, or you genuinely don't care to optimize.

Don't use it when: your documents have structure (sections, tables, code, lists). Fixed-size chunking will cleave through them indiscriminately.

Strategy 2: Sentence/paragraph-aware chunking

Use punctuation and paragraph breaks as the chunk boundaries. Combine sentences greedily up to a target token count, never breaking mid-sentence. Add the previous sentence as overlap.

This is the smallest behavior change from fixed-size that gives the biggest accuracy lift. ~5 lines of code with NLTK or a regex; 10–20% better recall on prose-heavy corpora in our internal benchmarks.

Use it when: your corpus is mostly written prose — blog posts, support tickets, transcripts, articles.

Strategy 3: Structural chunking

Respect document structure. For Markdown, split on H2/H3 headings. For HTML, split on section/article boundaries. For code, split on function/class boundaries. For tables, keep each table as one chunk (or one row per chunk for wide tables).

Each chunk is a semantically coherent unit — you never lose the heading that defines what a section is about, and you never split a code block.

Use it when: your documents have real structure — technical docs, code, API references, structured FAQs.

Strategy 4: Semantic / agentic chunking

The newest approach: an LLM (or a smaller classifier) reads the document and decides chunk boundaries based on topic shifts. "This sentence starts a new sub-topic; cut here."

Highest accuracy of any approach we've benchmarked — typically another 10–15% recall lift over structural chunking — but the ingest cost is non-trivial (one LLM call per ~5–10 pages).

Use it when: ingest cost is acceptable (you're processing thousands of documents once, not millions continuously), and retrieval accuracy matters more than per-document throughput.

The chunk size question

Everyone asks "what's the right chunk size?" and the honest answer is it depends on what the answers in your corpus look like. Rules of thumb:

  • Q&A / support / FAQ corpus: 256–512 tokens. Answers are typically short and discrete; small chunks keep retrieval precise.
  • Technical documentation: 512–1024 tokens. Sections and explanations need surrounding context.
  • Long-form articles / papers: 1024–2048 tokens. Arguments and citations need room to breathe.
  • Code: by function/class boundary, ignore token count. Functions vary; respect the semantic unit.

The 10–20% overlap rule is a starting point; tune it down toward 5% if you're seeing duplicate retrieval, up toward 25% if you're seeing answers cut across chunk boundaries.

The mistake every team makes once

Picking the chunk size without measuring. People copy 512 tokens from a blog post, ship, and never revisit. Then they spend three weeks tuning their reranker to fix a chunking problem.

Build a ~50-question eval set against your corpus, measure recall at top-K for two or three chunk-size + strategy combinations, pick the best one. The whole experiment takes a day and saves a month of pointless tuning downstream. See How to Evaluate a RAG Pipeline for the eval pattern.

Try chunking strategies on your own text

Our free Text Chunker lets you paste any document and instantly preview how it splits under each strategy. The chunk overlaps are visualized; you can A/B the same paragraph under fixed vs sentence vs structural in 30 seconds.

For production: the VertexRAG API ships agentic chunking out of the box — every document gets adaptive chunks that respect both structure and semantic boundaries, without you wiring an extra pipeline.

Read more on the VertexRAG blog or try the Graph RAG API.