Chapter 3: Chunking Long Documents - Velmurugan Narayanasamy

Every document in Chapters 1 and 2 was one sentence. That made the examples clean, but it quietly sidestepped something every real RAG system has to deal with: real documents are long. An employee handbook isn’t one sentence, it’s pages, with a vacation section, a sick leave section, an expense section, and more, all in one file.

So what happens if we just embed the whole handbook as a single vector, the same way we embedded single sentences in Chapter 2? Two problems show up immediately:

Length limits. Real embedding models accept a fixed maximum amount of text (typically somewhere from a few hundred to a few thousand words, depending on the model), then either truncate or refuse anything longer. A long document may not even fit.
Dilution. Even when it fits, averaging an entire multi-topic document into one vector blends the vacation section, the sick leave section, and the code of conduct into a single blurry point in space. A question about one specific detail has to compete against everything else in the document for that one vector’s “attention.”

Chapter 3 of 10 · Foundations

Where we left off: Chapter 2 fixed spelling-vs-meaning, but every document it compared was a single sentence.

What we fix now: Split long, multi-topic documents into smaller pieces before embedding them, so a question only has to compete against a focused chunk, not an entire document.

By the end, you can:

explain why embedding a whole multi-topic document as one vector causes dilution
build and compare three chunking strategies: whole-document, fixed-size, and sentence-aware
see how a badly placed chunk boundary can cut a fact in half, and fix it with overlap

The fix: chunk before you embed¶

The fix is the same shape as Chapter 2’s fix: change what gets compared. Instead of one vector per document, split each document into smaller pieces first, a chunk is just a contiguous slice of text, embed each chunk separately, and retrieve at the chunk level. The question now only has to compete against focused pieces of text, not entire documents.

The interesting part, and where real systems spend a lot of care, is how you cut a document into chunks. Cut carelessly and you can slice a fact in half right at a chunk boundary, which causes its own new failures. This chapter builds a few chunking strategies from scratch and shows exactly where each one helps and where it still breaks.

import numpy as np
import re

# Same co-occurrence + SVD technique as Chapter 2, extended with a few
# more topics so we have enough vocabulary for a longer document.
training_corpus = [
    # --- reused from Chapter 2 ---
    "Employees may carry over unused vacation days into the next calendar year.",
    "Staff can roll over unused holiday days into next year.",
    "Workers get paid time off and vacation days every year.",
    "Most employees take a vacation or holiday once a year.",
    "Unused vacation time can roll over to next year for many employees.",
    "Holiday leave and vacation leave are treated the same way at most companies.",
    "Some employees prefer to use their holiday days instead of saving them.",
    "Vacation days and holiday days both count as paid time off.",
    "Expense reports must be submitted within days of the purchase.",
    "Employees submit an expense report after every business purchase.",
    "The finance team reviews each expense report and each purchase receipt.",
    "A purchase over the limit needs approval before the expense is reimbursed.",
    "Expense claims and purchase receipts are reviewed together by finance.",
    "Remote employees receive a home office stipend to set up their workspace.",
    "The company gives remote workers a stipend for home office equipment.",
    "Employees working remote from home can request an office stipend.",
    "A stipend helps remote staff pay for office furniture at home.",
    "The office stipend is a one time payment for remote employees.",
    "Vacation policy, expense policy, and remote work policy are all in the handbook.",
    "Every employee should read the handbook for vacation and expense rules.",
    # --- new for Chapter 3 ---
    "Employees who are sick should notify their manager as soon as possible.",
    "Sick leave is paid time off for employees who are too unwell to work.",
    "A doctor's note may be required after three consecutive sick days.",
    "Unused sick leave does not roll over into the next year.",
    "Employees should stay home and rest when they are sick.",
    "Standard working hours are nine to five, Monday through Friday.",
    "Employees may request a flexible schedule with manager approval.",
    "Overtime must be approved in advance by a manager.",
    "Most staff work standard business hours unless a flexible schedule is approved.",
    "Employees are expected to treat colleagues and customers with respect.",
    "Harassment or discrimination of any kind will not be tolerated.",
    "Violations of the code of conduct may result in disciplinary action.",
    "Every employee must follow the code of conduct and treat others with respect.",
    "Company laptops must be encrypted and password protected.",
    "Employees should not install unauthorized software on company devices.",
    "Report a lost or stolen laptop to IT immediately.",
    "IT equipment must be returned when an employee leaves the company.",
    "The IT team manages laptops, software, and device security for the company.",
]

def tokenize(text):
    return re.findall(r"[a-z']+", text.lower())

tokens_per_sentence = [tokenize(s) for s in training_corpus]
vocab = sorted(set(w for sent in tokens_per_sentence for w in sent))
word_index = {w: i for i, w in enumerate(vocab)}
V = len(vocab)

window = 4
cooc = np.zeros((V, V))
for sent in tokens_per_sentence:
    for i, w in enumerate(sent):
        for j in range(max(0, i - window), min(len(sent), i + window + 1)):
            if i != j:
                cooc[word_index[w], word_index[sent[j]]] += 1

X = np.log1p(cooc)
U, S, Vt = np.linalg.svd(X, full_matrices=False)
DIM = 25
word_vectors = U[:, :DIM] * S[:DIM]

def embedding(word):
    return word_vectors[word_index[word]] if word in word_index else None

def embed_text(text):
    words = [w for w in tokenize(text) if w in word_index]
    if not words:
        return np.zeros(DIM)
    return np.mean([embedding(w) for w in words], axis=0)

def cosine_similarity(a, b):
    norm = np.linalg.norm(a) * np.linalg.norm(b)
    return float(np.dot(a, b) / norm) if norm else 0.0

print(f"Vocabulary size: {V} words")

# A realistic multi-section document: one company handbook,
# six policies in one file, the way it would actually exist.
handbook_document = """Vacation Policy
Employees may carry over up to 5 unused vacation days into the next calendar year. Time off should be requested from a manager at least two weeks in advance when possible.

Sick Leave Policy
Employees who are too unwell to work should notify their manager as soon as possible. Sick leave is paid time off and does not roll over into the next year. A doctor's note may be required after three consecutive sick days.

Expense Policy
Expense reports must be submitted within 30 days of the purchase date. The finance team reviews each expense report and matching purchase receipt before reimbursement.

Remote Work Policy
Remote employees are provided a one-time home office stipend of 500 euros. The stipend covers office furniture and basic equipment needed to work from home.

Working Hours
Standard working hours are nine to five, Monday through Friday. Employees may request a flexible schedule with manager approval, and any overtime must be approved in advance.

Code of Conduct
Employees are expected to treat colleagues and customers with respect. Harassment or discrimination of any kind will not be tolerated, and violations may result in disciplinary action."""

# A second, shorter, completely unrelated document, for comparison.
short_document = (
    "IT Equipment Policy: Company laptops must be encrypted and "
    "password protected. Employees should not install unauthorized "
    "software on company devices. Report a lost or stolen laptop to "
    "IT immediately."
)

question = "how many vacation days can be carried over into next year?"

print(f"Handbook length: {len(handbook_document)} characters, 6 sections")
print(f"Short document length: {len(short_document)} characters, 1 topic")

# The naive approach: embed each whole document as a single vector,
# exactly like Chapter 2 did with single sentences.
question_vector = embed_text(question)

for name, doc in [
    ("handbook (whole document, 6 sections)", handbook_document),
    ("IT equipment doc (short, single topic)", short_document),
]:
    score = cosine_similarity(question_vector, embed_text(doc))
    print(f"{score:.3f}  {name}")

The handbook still comes out on top here, but look at the margin: it’s not a landslide, even though the handbook contains the exact answer and the IT document has nothing to do with vacation days. Every irrelevant sentence in the handbook, sick leave, expense, working hours, code of conduct, pulls the whole-document vector a little further from “vacation,” diluting the one part that actually matters.

And even where this ranking happens to be correct, think about what you’d actually hand to a generation step: the entire six-section handbook, when the honest answer is one sentence. That’s wasted context for the model to sort through, and in a real system, wasted cost.

# Chunk by section instead: split on blank lines, embed each
# section separately, and rank the sections themselves.
section_chunks = [p.strip() for p in handbook_document.split("\n\n")]

ranked = sorted(
    section_chunks,
    key=lambda c: cosine_similarity(question_vector, embed_text(c)),
    reverse=True,
)
for c in ranked:
    score = cosine_similarity(question_vector, embed_text(c))
    title = c.splitlines()[0]
    print(f"{score:.3f}  {title}")

A large jump: the Vacation Policy section alone now scores well above the whole handbook’s 0.76, and well above the IT document too. More importantly, what you’d actually retrieve is two sentences about vacation days, not six sections of mostly-irrelevant policy. Splitting on blank lines worked well here because this handbook happens to have clean section breaks. Most real documents don’t hand you paragraph breaks that conveniently, so we need a strategy that doesn’t depend on getting lucky with formatting.

Before · whole document

Same question: “how many vacation days can be carried over into next year?”

Similarity score for the entire six-section handbook: 0.762. The right answer is buried somewhere inside it, alongside five sections that have nothing to do with the question.

After · chunked by section

Same question: “how many vacation days can be carried over into next year?”

Similarity score for the Vacation Policy section alone: 0.910, a clear jump above the whole document’s score. What gets retrieved now is two relevant sentences, not six sections.

Fixed-size chunking, and its obvious problem¶

The simplest general-purpose strategy: cut the text every N characters, no matter what’s there. It requires no assumptions about formatting, and it can slice a sentence, or a word, right down the middle. This chapter counts characters specifically to keep the mechanism visible; real RAG pipelines usually define chunk size in tokens instead, since tokens are the unit embedding models and LLM context windows actually operate on. Don’t assume a production chunk_size=500 means 500 characters, it almost always means tokens.

def fixed_chunks(text, chunk_size, overlap=0):
    chunks = []
    start = 0
    step = chunk_size - overlap
    while start < len(text):
        chunks.append(text[start:start + chunk_size])
        start += step
    return chunks

# A chunk size picked so the boundary lands inside the exact fact
# we care about, on purpose, to make the failure obvious.
chunks = fixed_chunks(handbook_document, chunk_size=63)
print("Chunk 0:", repr(chunks[0]))
print("Chunk 1:", repr(chunks[1]))

best = max(chunks, key=lambda c: cosine_similarity(question_vector, embed_text(c)))
best_score = cosine_similarity(question_vector, embed_text(best))
print(f"\nBest-scoring chunk: {best_score:.3f}")

has_full_fact = any("5 unused vacation days" in c for c in chunks)
print("Does any single chunk contain the complete phrase "
      f"'5 unused vacation days'? {has_full_fact}")

Look closely at chunk 0 and chunk 1: the word “vacation” is sliced into “vacatio” and “n” across the boundary. The similarity score barely notices, our small toy model is forgiving because other nearby words (“days,” “calendar,” “year”) still carry the meaning, but check the last line of output: no single chunk contains the complete fact “5 unused vacation days.” It’s split across two pieces. If a real system retrieves only the top-scoring chunk, whatever step reads it next, a person or a model, never sees the whole number in one place.

# The fix: let consecutive chunks overlap, so a fact sitting near a
# boundary has a chance to appear whole in at least one chunk.
chunks_overlap = fixed_chunks(handbook_document, chunk_size=63, overlap=20)

contains_full_fact = ["5 unused vacation days" in c for c in chunks_overlap]
for i, (c, has_it) in enumerate(zip(chunks_overlap, contains_full_fact)):
    if has_it:
        print(f"Chunk {i} contains the complete fact:")
        print(" ", repr(c))

print(f"\nAny chunk contains the complete phrase? {any(contains_full_fact)}")

Same chunk size, same document, one new parameter: overlap=20. Now at least one chunk contains “5 unused vacation days” fully intact, because the 20 characters right at the danger zone get repeated in two neighboring chunks instead of split between them. Overlap doesn’t prevent every possible bad cut, but it makes them far less likely to cost you a fact entirely.

The size trade-off, and a middle ground¶

Chunk size is a genuine trade-off, not a setting with one correct answer:

Too small, and you get the boundary problem above: facts and even single words get sliced apart, and each chunk carries too little surrounding context to be useful on its own.
Too large, and you’re sliding back toward the very first demo in this chapter: a chunk that spans multiple topics dilutes back down toward whole-document behavior, just with extra steps.

A common middle ground: sentence-aware chunking. Never cut in the middle of a sentence; instead, group whole sentences together up to a target size. It respects the one boundary that almost always matters (the sentence), while still giving you control over roughly how big each chunk is.

def sentence_chunks(text, target_size):
    sentences = re.split(r"(?<=[.!?])\s+", text.replace("\n\n", " ").strip())
    chunks, current = [], ""
    for s in sentences:
        if current and len(current) + len(s) + 1 > target_size:
            chunks.append(current.strip())
            current = s
        else:
            current = (current + " " + s).strip()
    if current:
        chunks.append(current.strip())
    return chunks

sentence_aware = sentence_chunks(handbook_document, target_size=220)
for c in sentence_aware:
    print("-", c[:70] + ("..." if len(c) > 70 else ""))

best = max(sentence_aware, key=lambda c: cosine_similarity(question_vector, embed_text(c)))
print(f"\nBest score: {cosine_similarity(question_vector, embed_text(best)):.3f}")

No word gets cut in half anywhere in this list, and the score is close to the section-chunking result from earlier. It’s not perfect, look closely and you’ll find at least one chunk that quietly crosses from one policy into the next, since “group sentences up to a size budget” doesn’t actually know what a “topic” is, only where sentences end. That’s a real, still-open limitation, not a bug to fix in this chapter.

CHUNK_SIZE = 63
OVERLAP = 20

my_chunks = fixed_chunks(handbook_document, CHUNK_SIZE, OVERLAP)
ranked = sorted(
    my_chunks,
    key=lambda c: cosine_similarity(question_vector, embed_text(c)),
    reverse=True,
)
for c in ranked[:5]:
    score = cosine_similarity(question_vector, embed_text(c))
    print(f"{score:.3f}  {c!r}")

Where it breaks

It still doesn’t scale by comparing one at a time. This chapter ranked a handful of chunks from one document. A real knowledge base might have millions of chunks across thousands of documents, computing cosine similarity against every single one for every question stops being fast, or cheap.
A chunk can lack context of its own. The winning chunk from the fixed-size demo above is a fragment: it doesn’t say which policy it’s from unless you also keep track of that separately. Some systems solve this by keeping a chunk’s surrounding text (or its section title) attached as metadata, so a narrow chunk still arrives with enough context to be useful.

Both of these point toward the same next question: once you have many chunks from many documents, how do you find the right ones fast, without comparing against everything?

Quick check¶

Before moving on, answer for yourself (no peeking ahead):

In the very first demo, the handbook still scored higher than the IT document, so the naive whole-document approach technically “worked.” What was still wrong with it, beyond the raw score?

Look at the two chunks produced by chunk_size=63 with no overlap. Why does neither one contain the complete phrase “5 unused vacation days,” and what specifically does adding overlap=20 change about how the chunks are built to fix that?

In the “try it yourself” cell, push CHUNK_SIZE up to something large, larger than an entire section. What happens to the winning chunk’s score, and why does that start to resemble the very first demo in this chapter?

Sentence-aware chunking never cuts a word or sentence in half, but it can still merge two different policies into a single chunk. Why might that still cause a problem for retrieval, even though nothing is technically “broken” the way the fixed-size cut was?

Chapter 2 pointed out that its toy embedding model has no vector at all for words it never saw during training. Does chunking make that specific problem better, worse, or unrelated? Explain.

You can now: split long documents into focused chunks and choose a strategy deliberately, whole-document, fixed-size with overlap, or sentence-aware, based on what each one actually trades off.

But one problem remains: we’ve now built retrieval three different ways, word overlap, embeddings, chunking, and never once actually generated an answer with a real model. It’s time to actually call one.

Next → Chapter 4: Completing the Loop