Chapter 5: Trustworthy Answers - Velmurugan Narayanasamy

Two loose ends, from two different chapters, turn out to be the same problem. Chapter 3 noticed that a retrieved chunk is a fragment: it doesn’t say which policy it came from unless you track that separately. Chapter 4 noticed that its demo printed an answer but never showed which chunk produced it. Both are really one gap: chunks carry no memory of where they came from.

The fix is simple to state: attach that information at chunking time, as data, and carry it through every step, retrieval, the prompt, generation, all the way to what the reader actually sees.

Chapter 5 of 10 · Building a grounded system

Where we left off: Chapter 3 noticed a retrieved chunk doesn’t say which policy it came from. Chapter 4 noticed its demo never showed which chunk produced an answer.

What we fix now: Attach provenance to every chunk as data, and carry it through retrieval, the prompt, and generation, all the way to what the reader actually sees.

By the end, you can:

explain what a citation does and does not prove about an answer
attach and carry metadata through a retrieval pipeline so mistakes become visible instead of invisible
build parent-document retrieval: search a precise fragment, display a clean full section

import numpy as np
import re

# Identical setup to Chapters 2-4, condensed.
training_corpus = [
    "Employees may carry over unused vacation days into the next calendar year.",
    "Staff can roll over unused holiday days into next year.",
    "Workers get paid time off and vacation days every year.",
    "Most employees take a vacation or holiday once a year.",
    "Unused vacation time can roll over to next year for many employees.",
    "Holiday leave and vacation leave are treated the same way at most companies.",
    "Some employees prefer to use their holiday days instead of saving them.",
    "Vacation days and holiday days both count as paid time off.",
    "Expense reports must be submitted within days of the purchase.",
    "Employees submit an expense report after every business purchase.",
    "The finance team reviews each expense report and each purchase receipt.",
    "A purchase over the limit needs approval before the expense is reimbursed.",
    "Expense claims and purchase receipts are reviewed together by finance.",
    "Remote employees receive a home office stipend to set up their workspace.",
    "The company gives remote workers a stipend for home office equipment.",
    "Employees working remote from home can request an office stipend.",
    "A stipend helps remote staff pay for office furniture at home.",
    "The office stipend is a one time payment for remote employees.",
    "Vacation policy, expense policy, and remote work policy are all in the handbook.",
    "Every employee should read the handbook for vacation and expense rules.",
    "Employees who are sick should notify their manager as soon as possible.",
    "Sick leave is paid time off for employees who are too unwell to work.",
    "A doctor's note may be required after three consecutive sick days.",
    "Unused sick leave does not roll over into the next year.",
    "Employees should stay home and rest when they are sick.",
    "Standard working hours are nine to five, Monday through Friday.",
    "Employees may request a flexible schedule with manager approval.",
    "Overtime must be approved in advance by a manager.",
    "Most staff work standard business hours unless a flexible schedule is approved.",
    "Employees are expected to treat colleagues and customers with respect.",
    "Harassment or discrimination of any kind will not be tolerated.",
    "Violations of the code of conduct may result in disciplinary action.",
    "Every employee must follow the code of conduct and treat others with respect.",
    "Company laptops must be encrypted and password protected.",
    "Employees should not install unauthorized software on company devices.",
    "Report a lost or stolen laptop to IT immediately.",
    "IT equipment must be returned when an employee leaves the company.",
    "The IT team manages laptops, software, and device security for the company.",
]

def tokenize(text):
    return re.findall(r"[a-z']+", text.lower())

tokens_per_sentence = [tokenize(s) for s in training_corpus]
vocab = sorted(set(w for sent in tokens_per_sentence for w in sent))
word_index = {w: i for i, w in enumerate(vocab)}
V = len(vocab)
window = 4
cooc = np.zeros((V, V))
for sent in tokens_per_sentence:
    for i, w in enumerate(sent):
        for j in range(max(0, i - window), min(len(sent), i + window + 1)):
            if i != j:
                cooc[word_index[w], word_index[sent[j]]] += 1
X = np.log1p(cooc)
U, S, Vt = np.linalg.svd(X, full_matrices=False)
DIM = 25
word_vectors = U[:, :DIM] * S[:DIM]

def embedding(word):
    return word_vectors[word_index[word]] if word in word_index else None

def embed_text(text):
    words = [w for w in tokenize(text) if w in word_index]
    if not words:
        return np.zeros(DIM)
    return np.mean([embedding(w) for w in words], axis=0)

def cosine_similarity(a, b):
    norm = np.linalg.norm(a) * np.linalg.norm(b)
    return float(np.dot(a, b) / norm) if norm else 0.0

handbook_document = """Vacation Policy
Employees may carry over up to 5 unused vacation days into the next calendar year. Time off should be requested from a manager at least two weeks in advance when possible.

Sick Leave Policy
Employees who are too unwell to work should notify their manager as soon as possible. Sick leave is paid time off and does not roll over into the next year. A doctor's note may be required after three consecutive sick days.

Expense Policy
Expense reports must be submitted within 30 days of the purchase date. The finance team reviews each expense report and matching purchase receipt before reimbursement.

Remote Work Policy
Remote employees are provided a one-time home office stipend of 500 euros. The stipend covers office furniture and basic equipment needed to work from home.

Working Hours
Standard working hours are nine to five, Monday through Friday. Employees may request a flexible schedule with manager approval, and any overtime must be approved in advance.

Code of Conduct
Employees are expected to treat colleagues and customers with respect. Harassment or discrimination of any kind will not be tolerated, and violations may result in disciplinary action."""

section_chunks = [p.strip() for p in handbook_document.split("\n\n")]
print(f"Ready: {len(section_chunks)} sections, {V} words in vocabulary.")

# Instead of bare strings, each chunk is now a small record that
# carries its own provenance wherever it goes.
records = []
for i, chunk in enumerate(section_chunks):
    records.append({
        "text": chunk,
        "section": chunk.splitlines()[0],
        "source": "Employee Handbook",
        "chunk_id": i,
    })

def retrieve_best(question, recs=records):
    return max(recs, key=lambda r: cosine_similarity(embed_text(question), embed_text(r["text"])))

print(records[0])

# A question this handbook genuinely answers, but not with the
# section you might expect.
question = "how many sick days can be carried over to next year?"

best = retrieve_best(question)
score = cosine_similarity(embed_text(question), embed_text(best["text"]))
print(f"Retrieved section: {best['section']!r}  (score {score:.3f})")

Retrieval confidently returns the Vacation Policy, not the Sick Leave Policy, for a question specifically about sick days. Look back at the handbook text: sick leave and vacation share almost identical vocabulary in this corpus (both about “days,” “roll over,” “paid time off,” “employees”), so our small embedding model, which only measures topical closeness, can’t tell “these days carry over” apart from “these days explicitly do not.” That’s the same single-vector limitation Chapter 2 flagged early on, showing up again in a new, more consequential place.

If a generation step answered using only this retrieved chunk, it would have every reason to say something like “up to 5 days,” the vacation number, applied to a question about sick leave, which the real Sick Leave Policy explicitly contradicts. That’s not a hypothetical: it’s exactly the kind of fluent, plausible, wrong answer Chapter 1 opened with.

The fix: show the source, not just the answer¶

Nothing about the retrieval step changes here, embeddings still have the limitation above, but now every retrieved record already carries what’s needed to make that limitation visible instead of invisible.

def build_prompt(question, context, refuse_if_missing=True):
    instructions = (
        "Answer the question using only the context below. "
        "If the answer is not contained in the context, say "
        "\"I don't know based on the provided context\" instead of guessing."
        if refuse_if_missing else
        "Answer the question using the context below."
    )
    return f"""{instructions}

Context:
{context}

Question:
{question}"""

def format_with_citation(answer_text, source_record):
    return (
        f"Answer: {answer_text}\n"
        f"Source: {source_record['section']} ({source_record['source']})"
    )

prompt = build_prompt(question, best["text"])
print(prompt)
print()
print("--- Citation that will accompany whatever answer comes back ---")
print(f"Source: {best['section']} ({best['source']})")

Read that citation on its own, before even seeing a generated answer: “Source: Vacation Policy.” Someone who asked about sick days and sees that source line has enough information right there to be skeptical, without needing to fact-check the answer’s wording at all. That’s the entire value of a citation: it doesn’t make retrieval more accurate, it makes retrieval’s mistakes visible instead of invisible. Chapter 1’s original complaint about ungrounded models was that a wrong answer reads exactly like a right one. A citation is a small, cheap way to break that disguise.

Parent-document retrieval: precise match, readable context¶

Chapter 3’s fixed-size chunking cut words in half at chunk boundaries in the name of simplicity. Metadata fixes that problem too: search using small, precise fragments, but keep a link back to the full, readable section each fragment came from, and hand that to the reader or the model instead of the choppy fragment itself.

def fixed_chunks_with_parent(document, sections, chunk_size, overlap=0):
    # Find where each section starts and ends in the full document,
    # so every small fragment can be traced back to its full parent.
    section_spans = []
    pos = 0
    for s in sections:
        start = document.index(s, pos)
        end = start + len(s)
        section_spans.append((start, end, s))
        pos = end

    frags = []
    step = chunk_size - overlap
    start = 0
    while start < len(document):
        piece = document[start:start + chunk_size]
        midpoint = start + len(piece) // 2
        parent = next((sec for (s, e, sec) in section_spans if s <= midpoint < e), None)
        frags.append({
            "text": piece,
            "parent_text": parent,
            "section": parent.splitlines()[0] if parent else None,
        })
        start += step
    return frags

fine_grained = fixed_chunks_with_parent(handbook_document, section_chunks, chunk_size=63)

vacation_question = "how many vacation days can be carried over into next year?"
best_fragment = max(
    fine_grained,
    key=lambda r: cosine_similarity(embed_text(vacation_question), embed_text(r["text"])),
)
print("Fragment actually matched (this is what search compared against):")
print(" ", repr(best_fragment["text"]))
print()
print("Parent section handed to the reader instead:")
print(best_fragment["parent_text"])

The fragment that won the search is choppy, it cuts off mid-word, exactly Chapter 3’s boundary problem. Nobody should read that fragment directly. But because it carries a link to its parent section, what actually gets shown is the full, clean paragraph, precise matching underneath, readable context on top.

my_question = "what happens if I lose my company laptop?"

best_record = retrieve_best(my_question)
score = cosine_similarity(embed_text(my_question), embed_text(best_record["text"]))
print(f"Retrieved section: {best_record['section']!r}  (score {score:.3f})")
print(f"Citation: Source: {best_record['section']} ({best_record['source']})")

Bonus (Colab only): a real answer, with its real citation side by side¶

Same limitation as every chapter so far: this needs a live network call and an API key, neither available in this page. Open in Colab to run it, using a free Gemini key from aistudio.google.com/apikey:

# Colab only: needs a live network call and an API key.
!pip install -q -U google-genai

from google import genai

API_KEY = "YOUR_API_KEY_HERE"
client = genai.Client(api_key=API_KEY)
MODEL = "gemini-3.5-flash"

sick_question = "how many sick days can be carried over to next year?"
retrieved = retrieve_best(sick_question)
prompt = build_prompt(sick_question, retrieved["text"])

response = client.models.generate_content(model=MODEL, contents=prompt)

print(format_with_citation(response.text, retrieved))

Read the answer and the source line together. If the answer states a specific number of carried-over days, and the source line says “Vacation Policy” for a question about sick leave, that mismatch is now something the reader can catch in two seconds, instead of a wrong fact quietly absorbed as truth.

Where it breaks

A citation is only as good as its metadata. Everything here depends on section and source being recorded correctly when chunks are built. Wrong metadata produces a confidently wrong citation, which is arguably worse than no citation, since it looks trustworthy.
A citation shows which retrieved source accompanied the answer, not that the answer is correct, or even that the model actually used it. A model could in principle ignore the provided context entirely and a citation would still be displayed right alongside its answer. Even when retrieval hands over the right section, a model can still misread or misstate what’s in it. Citations make answers checkable; they don’t make them automatically right.
This still doesn’t scale. Every chapter so far has compared a question against a handful of chunks directly. A real system with millions of chunks across thousands of documents still needs a way to find candidates fast, without comparing against everything, the same open thread since Chapter 4.

Quick check¶

Before moving on, answer for yourself (no peeking ahead):

Why does attaching metadata to a chunk not fix the underlying retrieval mistake (Vacation Policy still gets retrieved for a sick leave question)? What does it actually fix instead?

In the parent-document demo, the system searched using a choppy, mid-word fragment but displayed a clean full section. Why search with the fragment at all, instead of just always searching with full sections like earlier chapters did?

Suppose a chunk’s section metadata was recorded incorrectly by a bug somewhere upstream. Would you rather have no citation at all, or a citation that’s wrong? Why?

This chapter shows a citation next to an answer. What would you still need to check, as a reader, even after seeing a citation that looks correct?

Chapter 4 built a refuse-if-missing instruction so the model says “I don’t know” when context doesn’t answer the question. Does adding citations, this chapter’s fix, make that instruction less necessary, more necessary, or unrelated? Explain with a concrete scenario.

You can now: attach real provenance to every chunk and carry it through retrieval, the prompt, and the final answer, so a retrieval mistake becomes visible instead of invisible.

But one problem remains: every chapter so far has compared a question against a handful of chunks directly. A real system with millions of chunks still needs a way to find candidates fast, without comparing against everything.

Next → Chapter 6: Scaling Retrieval