Chapter 4: Completing the Loop - Velmurugan Narayanasamy

Chapter 1 defined RAG as two steps: retrieve, then generate. Three chapters later, we’ve built retrieve three different ways, word overlap, embeddings, chunking, and never once actually generated anything. Chapter 1 showed what a model would receive; it never called one. This chapter finishes the loop.

This chapter uses Google’s Gemini API for the live demo near the end. It has a genuine free tier: no credit card required, no expiring trial credit, just a free API key from aistudio.google.com/apikey.

Chapter 4 of 10 · Building a grounded system

Where we left off: We’ve built retrieval three different ways now, word overlap, embeddings, chunking, and never once actually generated an answer with a real model.

What we fix now: Actually call a language model: assemble a grounded prompt from the retrieved context and the question, and get a real generated answer from a real, live API.

By the end, you can:

explain the grounding contract, the instruction that does almost all the work in this chapter
build a prompt that refuses to answer when the retrieved context doesn’t contain the answer
see, with a real live model call, that the instruction is a request, not a guarantee

Setup: reuse Chapters 2 and 3 as a black box¶

The next cell rebuilds the retrieval pipeline from Chapters 2 and 3 (the co-occurrence embedding model, the handbook, section-based chunking) without re-explaining it, since that’s already covered. Treat it as a known-working function: give it a question, get back the best-matching chunk.

import numpy as np
import re

training_corpus = [
    "Employees may carry over unused vacation days into the next calendar year.",
    "Staff can roll over unused holiday days into next year.",
    "Workers get paid time off and vacation days every year.",
    "Most employees take a vacation or holiday once a year.",
    "Unused vacation time can roll over to next year for many employees.",
    "Holiday leave and vacation leave are treated the same way at most companies.",
    "Some employees prefer to use their holiday days instead of saving them.",
    "Vacation days and holiday days both count as paid time off.",
    "Expense reports must be submitted within days of the purchase.",
    "Employees submit an expense report after every business purchase.",
    "The finance team reviews each expense report and each purchase receipt.",
    "A purchase over the limit needs approval before the expense is reimbursed.",
    "Expense claims and purchase receipts are reviewed together by finance.",
    "Remote employees receive a home office stipend to set up their workspace.",
    "The company gives remote workers a stipend for home office equipment.",
    "Employees working remote from home can request an office stipend.",
    "A stipend helps remote staff pay for office furniture at home.",
    "The office stipend is a one time payment for remote employees.",
    "Vacation policy, expense policy, and remote work policy are all in the handbook.",
    "Every employee should read the handbook for vacation and expense rules.",
    "Employees who are sick should notify their manager as soon as possible.",
    "Sick leave is paid time off for employees who are too unwell to work.",
    "A doctor's note may be required after three consecutive sick days.",
    "Unused sick leave does not roll over into the next year.",
    "Employees should stay home and rest when they are sick.",
    "Standard working hours are nine to five, Monday through Friday.",
    "Employees may request a flexible schedule with manager approval.",
    "Overtime must be approved in advance by a manager.",
    "Most staff work standard business hours unless a flexible schedule is approved.",
    "Employees are expected to treat colleagues and customers with respect.",
    "Harassment or discrimination of any kind will not be tolerated.",
    "Violations of the code of conduct may result in disciplinary action.",
    "Every employee must follow the code of conduct and treat others with respect.",
    "Company laptops must be encrypted and password protected.",
    "Employees should not install unauthorized software on company devices.",
    "Report a lost or stolen laptop to IT immediately.",
    "IT equipment must be returned when an employee leaves the company.",
    "The IT team manages laptops, software, and device security for the company.",
]

def tokenize(text):
    return re.findall(r"[a-z']+", text.lower())

tokens_per_sentence = [tokenize(s) for s in training_corpus]
vocab = sorted(set(w for sent in tokens_per_sentence for w in sent))
word_index = {w: i for i, w in enumerate(vocab)}
V = len(vocab)
window = 4
cooc = np.zeros((V, V))
for sent in tokens_per_sentence:
    for i, w in enumerate(sent):
        for j in range(max(0, i - window), min(len(sent), i + window + 1)):
            if i != j:
                cooc[word_index[w], word_index[sent[j]]] += 1
X = np.log1p(cooc)
U, S, Vt = np.linalg.svd(X, full_matrices=False)
DIM = 25
word_vectors = U[:, :DIM] * S[:DIM]

def embedding(word):
    return word_vectors[word_index[word]] if word in word_index else None

def embed_text(text):
    words = [w for w in tokenize(text) if w in word_index]
    if not words:
        return np.zeros(DIM)
    return np.mean([embedding(w) for w in words], axis=0)

def cosine_similarity(a, b):
    norm = np.linalg.norm(a) * np.linalg.norm(b)
    return float(np.dot(a, b) / norm) if norm else 0.0

handbook_document = """Vacation Policy
Employees may carry over up to 5 unused vacation days into the next calendar year. Time off should be requested from a manager at least two weeks in advance when possible.

Sick Leave Policy
Employees who are too unwell to work should notify their manager as soon as possible. Sick leave is paid time off and does not roll over into the next year. A doctor\'s note may be required after three consecutive sick days.

Expense Policy
Expense reports must be submitted within 30 days of the purchase date. The finance team reviews each expense report and matching purchase receipt before reimbursement.

Remote Work Policy
Remote employees are provided a one-time home office stipend of 500 euros. The stipend covers office furniture and basic equipment needed to work from home.

Working Hours
Standard working hours are nine to five, Monday through Friday. Employees may request a flexible schedule with manager approval, and any overtime must be approved in advance.

Code of Conduct
Employees are expected to treat colleagues and customers with respect. Harassment or discrimination of any kind will not be tolerated, and violations may result in disciplinary action."""

section_chunks = [p.strip() for p in handbook_document.split("\n\n")]

def retrieve_best(question, chunks=section_chunks):
    return max(chunks, key=lambda c: cosine_similarity(embed_text(question), embed_text(c)))

print(f"Ready: {len(section_chunks)} chunks, {V} words in vocabulary.")

# One question the handbook actually answers, and one it doesn't.
question_real = "how many vacation days can be carried over into next year?"
question_out_of_scope = "does the company offer pet insurance?"

for label, q in [("real question", question_real), ("out-of-scope question", question_out_of_scope)]:
    best = retrieve_best(q)
    score = cosine_similarity(embed_text(q), embed_text(best))
    print(f"{label}: {q!r}")
    print(f"  best chunk ({score:.3f}): {best.splitlines()[0]!r}\n")

Notice: retrieval “worked” for both questions, in the sense that it confidently returned a chunk each time. For the pet insurance question, that confidence is misleading. Nothing in this handbook is about pet insurance, but cosine similarity always returns something, whichever chunk happens to score highest, even if the true answer is “not in here at all.”

Our retriever here has no mechanism for returning “none of the above”: it always hands back its best-scoring chunk, confident or not. Production systems often add exactly that mechanism on the retrieval side instead, a relevance threshold, a separate verifier or classifier model, metadata filtering, so a genuinely irrelevant match never reaches generation in the first place. This simple pipeline doesn’t build one of those. Instead, we’ll rely entirely on the generate step: instruct the model to say it doesn’t know when the supplied context doesn’t answer the question, and see how far that gets us on its own.

Building the actual prompt¶

Chapter 1 showed roughly what a generate step receives. Here’s the real version: a function that takes a question and a retrieved chunk and assembles the exact text that would be sent to a model, plus one explicit instruction that does almost all of the work in this chapter.

def build_prompt(question, context, refuse_if_missing=True):
    if refuse_if_missing:
        instructions = (
            "Answer the question using only the context below. "
            "If the answer is not contained in the context, say "
            "\"I don\'t know based on the provided context\" instead of guessing."
        )
    else:
        instructions = "Answer the question using the context below."

    return f"""{instructions}

Context:
{context}

Question:
{question}"""

context_for_real_q = retrieve_best(question_real)
prompt_real = build_prompt(question_real, context_for_real_q)
print(prompt_real)

That refuse_if_missing instruction is the entire grounding contract. There’s no special mechanism making a model “know” what it doesn’t know, it can’t inspect its own certainty. All we can do is explicitly tell it what to do when the context doesn’t contain an answer, and hope it follows that instruction. That’s a real limitation to sit with before moving on: it’s an instruction, not a guarantee.

question = question_out_of_scope
refuse_if_missing = True

context = retrieve_best(question)
prompt = build_prompt(question, context, refuse_if_missing)
print(prompt)

Look at what got retrieved for this question: a real section of the handbook, confidently handed over, that has nothing to do with pet insurance. The instruction above is the only thing standing between that mismatch and a model inventing a policy that doesn’t exist.

Bonus (Colab only): a real generated answer¶

Everything above runs anywhere, it’s just string building. Actually calling a model needs a live API call, which needs real network access and an API key, neither of which this in-browser page can provide (the same runnable-but-not-editable limitation as every previous chapter). Open this notebook in Colab to run the cell below:

Get a free key at aistudio.google.com/apikey (no credit card needed), then paste it into API_KEY below. Treat it like a password: fine to paste into your own private Colab copy while experimenting, never commit it to a public repository or notebook.

# Colab only: needs a live network call and an API key.
!pip install -q -U google-genai

from google import genai

API_KEY = "YOUR_API_KEY_HERE"
client = genai.Client(api_key=API_KEY)
MODEL = "gemini-3.5-flash"

def ask(prompt):
    response = client.models.generate_content(model=MODEL, contents=prompt)
    return response.text

for label, q in [("real question", question_real), ("out-of-scope question", question_out_of_scope)]:
    context = retrieve_best(q)
    grounded_prompt = build_prompt(q, context, refuse_if_missing=True)
    ungrounded_prompt = q  # the same question, no retrieved context at all

    print(f"=== {label}: {q!r} ===")
    print("Grounded (retrieved context + refuse-if-missing instruction):")
    print(" ", ask(grounded_prompt))
    print("Ungrounded (just the raw question, nothing retrieved):")
    print(" ", ask(ungrounded_prompt))
    print()

For the real question, both answers are probably fine, the model likely already “knows” a plausible-sounding vacation policy from training data, which is exactly Chapter 1’s original problem: a fluent, plausible-sounding answer that may or may not match your actual policy. The grounded answer is the one that’s actually correct, provably, because you can point at the exact sentence it came from.

For the out-of-scope question, watch for a real difference: the grounded version, if the instruction is followed, should say something like “I don’t know based on the provided context.” The ungrounded version has nothing stopping it from inventing a pet insurance policy that sounds entirely plausible and does not exist. Run it a few times; model behavior isn’t perfectly consistent, which is itself worth noticing.

Where it breaks

Garbage in, garbage out. The grounded answer is only as good as the retrieved chunk. Feed it the wrong chunk (like the Expense Policy for a pet insurance question) and even a well-instructed model has bad raw material to work with.
The instruction isn’t bulletproof. “Say I don’t know” is a request, not a constraint the model is physically incapable of violating. Models sometimes answer anyway, especially when a retrieved chunk is topically close but doesn’t actually contain the answer.
Every answer costs something. Calling a real model for every question has real latency and, outside a free tier, real cost. A production system answering thousands of questions a day has to think about caching, batching, and when it’s even worth calling a large model versus a smaller one.
No source is shown to the reader. The demo above prints an answer, but not which chunk it came from. A trustworthy system usually shows its source alongside the answer, so a person can go check it, which is the entire reason RAG is more checkable than a model answering from memory in the first place.

Quick check¶

Before moving on, answer for yourself (no peeking ahead):

In your own words, what is the “grounding contract,” and where exactly does it live? Is it a property of the model, the retrieval step, or something else?

Retrieval returned a chunk for the pet insurance question with a reasonably high similarity score. Why doesn’t a high similarity score guarantee the retrieved chunk actually answers the question?

If you ran the Colab cell multiple times on the out-of-scope question, did you get the same answer every time? What does that tell you about relying on an instruction like “say I don’t know” as your only safeguard?

This chapter’s demo has no way to show the reader which chunk an answer came from. Why does that matter more for a RAG system than it would for a model just answering from what it learned in training?

Chapter 1 showed that if retrieval hands the wrong document to generation, generation can’t recover. This chapter’s refuse-if-missing instruction doesn’t fix that Chapter 1 problem, so what problem does it actually solve, and which is the more fundamental limitation: a wrong chunk being retrieved, or a model failing to say “I don’t know”?

You can now: actually call a real model with a grounded prompt, and you’ve seen that the refuse-if-missing instruction is a request the model can still ignore.

But one problem remains: the demo prints an answer but never shows which chunk it came from, so a reader has no way to check it, exactly the property that’s supposed to make RAG answers trustworthy in the first place.

Next → Chapter 5: Trustworthy Answers