R and D · 2026-03-10 · Go Kyono
A comparative study of text chunking methods
Chunking is the preprocessing step that sets the ceiling on retrieval quality, and it is usually chosen by habit. This note compares the five common methods across five axes and reports where chunk size and overlap actually landed across several domains.
Short answers first
If you only came for one definition
- What is chunking?
- Splitting long documents into coherent units before indexing them, so retrieval can return the relevant passage rather than the whole document. The quality of the split sets the ceiling on both precision and recall, and therefore on answer quality.
- What is the difference between a chunk and a token?
- A token is a unit of length; a chunk is a piece of a document. Tokens are how a model counts text. A chunk is one unit of retrieval, whose size is expressed in tokens — 256 to 512, for example. The token is the ruler; the chunk is what you cut with it.
- Which method splits on meaning?
- Semantic chunking. Adjacent sentences or paragraphs are embedded, cosine similarity is computed between neighbors, and a boundary is placed where similarity drops below a threshold. It cuts on meaning rather than on length or formatting, and it is substantially slower than fixed-length because of the embedding cost.
- What is overlap for?
- Preserving context that straddles a boundary: text near the edge is included in both neighboring chunks, so a thought cut mid-argument is still retrievable whole. Overlap does not save space and does not remove redundancy — it deliberately adds redundancy, so the index gets larger. Ten to twenty percent of chunk size is the usual setting.
- What does recursive chunking buy over fixed-length?
- It avoids cutting through words and paragraphs. Separators are applied in priority order: paragraph first, then sentence, then word, descending only when a chunk exceeds the target size. Natural structure survives. Fixed-length cuts at whatever character the counter reaches, including the middle of a sentence.
- What chunk size should I start with?
- 256 to 512 tokens gave the best balance of retrieval accuracy and generation quality across the domains we tested. Start around 512 and adjust. Technical and legal documents, dense with defined terms and quoted clauses, did better at 512 to 768. Overlap of 10 to 15 percent helped; beyond 20 percent, redundancy degraded performance.
01
The five methods
Fixed-length
Split at a predetermined character or token count. Trivial to implement and very fast. It cuts through sentences and paragraphs, so semantic coherence is the price.
Fixed-length with overlap
The same, with a repeated region between neighbors. Context near a boundary survives, and retrieval accuracy improves accordingly. The index grows by the overlap ratio, which is the trade.
Semantic
Embed adjacent fragments, compute cosine similarity, and place a boundary where similarity falls below a threshold. Produces coherent chunks. Adds an embedding pass over the entire corpus, so ingestion time increases sharply.
Recursive
Apply separators in priority order — paragraph break, then sentence, then word — descending only when the result exceeds the target size. Respects natural document structure while keeping chunk size in a bounded range. The best default.
Document-structure based
Split on headings, sections and lists in Markdown, HTML or PDF. Each chunk is semantically self-contained because the author already made it so. Requires the document to have real structure, so it does not apply to plain text.
| Method | Coherence | Speed | Ease | Size uniformity | Format dependence |
|---|---|---|---|---|---|
| Fixed-length | Low | Very fast | Easy | High | None |
| Fixed with overlap | Slightly low | Fast | Easy | High | None |
| Semantic | High | Slow | Moderate | Low | None |
| Recursive | Medium to high | Fast | Moderate | Medium | Low |
| Structure-based | High | Fast | Moderate to hard | Low | High |
Every method carries a trade-off; none dominates. If coherence is the priority, semantic wins and you pay in ingestion time. If you want the best balance of speed and quality, recursive is the practical answer.
02
Chunk size
Chunk size is a balance between two opposing pressures.
Small chunks, up to about 256 tokens
- Finer retrieval granularity, so precision tends to improve
- Good for extracting a specific fact or figure
- Context can be too thin for the model to answer well
- More chunks means a larger index and higher search cost
Large chunks, 1024 tokens and above
- Richer context, so generation is more stable
- Fewer chunks, so the index is more efficient
- Irrelevant material rides along, and recall degrades
- Consumes context window that other passages needed
What we measured
Across several domains, evaluating retrieval and generation at a range of chunk sizes:
- 256 to 512 tokens was the best general-purpose range. It is close to a natural paragraph, which is presumably why coherence holds up.
- Domain shifts the optimum. Technical and legal text, dense with defined terms and quoted clauses, did better at 512 to 768.
- Question granularity correlates. Factoid questions favored smaller chunks; summary and explanation questions favored larger ones.
- Overlap of 10 to 15 percent reduced information loss at boundaries. Beyond 20 percent, the added redundancy degraded performance.
A procedure that works
- Set a baseline around 512 tokens.
- Analyze the expected question granularity and the document type, and adjust.
- Add 10 to 15 percent overlap.
- Score against a representative query set, and iterate.
03
Choosing a method
| Situation | Use |
|---|---|
| Prototype or validation | Recursive. Low implementation cost, sufficient quality. Fixed-with-overlap is an acceptable substitute |
| Production, accuracy critical | Semantic, or semantic combined with structure-based. Ingestion costs more; retrieval quality improves |
| Mostly structured documents | Structure-based first, falling back to recursive where structure is absent |
| High-volume, throughput bound | Fixed with overlap. Best speed-to-quality ratio at scale |
Where this is heading
- Adaptive chunking — switching size and method per document based on content and structure
- Hierarchical chunking — indexing at coarse and fine granularity simultaneously, selecting at query time
- Multimodal — extending chunking to documents containing tables, figures and equations
- Model-assisted boundaries — using an LLM to place the cuts. Accurate, and the cost is the open question
Chunking is unglamorous infrastructure, and optimizing it is one of the few levers that improves every query rather than a class of queries. It is also downstream of terminology: a beautifully cut index of inconsistent text is still an index of inconsistent text. That problem is covered in Ontology design in practice.
Next
We will measure yours.
Chunking strategy and retrieval accuracy work, scored against a gold set from your own domain rather than a public benchmark.