Chapter 1: What is RAG? - Velmurugan Narayanasamy

Imagine you ask an AI assistant something very specific to your situation, like “how many vacation days do I roll over into next year?”, “what’s the return policy on this specific product?”, “what does section 4.2 of this contract actually commit us to?”

A general-purpose language model has never seen your company’s leave policy, that product’s manual, or that contract. It wasn’t trained on your documents; it was trained on a huge slice of the public internet, frozen at some point in the past. So when you ask it something specific to your world, it does the only thing it can: it guesses based on what similar documents usually say. Sometimes that guess is close. Often it’s confidently, plausibly, invisibly wrong, and “invisibly” is the dangerous part, because a wrong answer delivered fluently reads exactly like a right one.

Chapter 1 of 10 · Foundations

Where we left off: Nowhere yet, this is the first chapter. Nothing to leave behind.

What we fix now: See why a general-purpose model struggles with your own documents, and build the simplest possible version of the retrieve-then-generate fix.

By the end, you can:

explain the two steps of RAG, retrieve and generate, and what each one is responsible for
build a word-matching retriever from scratch and use it
identify exactly why word-matching fails on a paraphrase like “holiday” vs “vacation”

The fix: retrieve, then generate¶

Retrieval-Augmented Generation (RAG) is a simple change to that picture: before the model answers, it’s given a chance to look something up. Two steps, every single time:

Retrieve: search a collection of your actual documents for the parts relevant to the question.
Generate: hand those retrieved parts to the model along with the question, so it answers from what it just read, not from whatever it half-remembers from training.

That’s the entire idea. No retraining the model, no fine-tuning, no teaching it new facts permanently, just giving it better material to work with, one question at a time. Here’s the same two steps as a picture:

Notice the question feeds into both steps: once to find the right material, once again alongside that material so the model has something to reason over. Skip the retrieve step and you’re back to a model guessing from memory. Skip the generate step and you just have a search engine, not something that can actually answer in plain language.

documents = [
    "Employees may carry over up to 5 unused vacation days into the next calendar year.",
    "Expense reports must be submitted within 30 days of the purchase date.",
    "Remote employees are provided a one-time home office stipend of 500 euros.",
]

question = "how many vacation days can I carry over into next year?"

def score(doc, question):
    doc_words = set(doc.lower().split())
    q_words = set(question.lower().split())
    return len(doc_words & q_words)

ranked = sorted(documents, key=lambda d: score(d, question), reverse=True)

print("Best match:", ranked[0])
print("Shared words:", score(ranked[0], question))

What “generate” actually receives¶

It’s worth being concrete about step 2, since it’s easy to leave as an abstraction. Once retrieval picks a document, the model doesn’t just get the raw question again; it gets a prompt built from both pieces, roughly like this:

Answer the question using only the context below.

Context:
"Employees may carry over up to 5 unused vacation days into the next
calendar year."

Question:
How many vacation days can I carry over into next year?

The model isn’t reciting a memorized policy: it’s reading the same sentence you’d read, and answering from it. That’s the entire trick behind why RAG answers can be checked: the source text is right there, not buried inside billions of model parameters.

question = "do unused holidays roll over to next year?"

ranked = sorted(documents, key=lambda d: score(d, question), reverse=True)

print("Best match:", ranked[0])
print("Shared words:", score(ranked[0], question))

“Holidays” and “vacation days” mean the same thing to a person. To this retriever, they share zero letters in common, so the score can come out low or even tied with a document that has nothing to do with the question. If two documents tie, or the right one scores lower than a wrong one, the generate step never gets a chance: it’s stuck answering from whatever retrieval handed it, wrong document included.

This is the exact gap embeddings are built to close: a way of comparing meaning instead of spelling, so “holidays” and “vacation days” land close together even though they don’t share a single word. That’s the next big idea in this series.

Quick check¶

Before moving on, answer for yourself (no peeking ahead):

What are the two steps in RAG, in order, and what does each one do?

Why does a general-purpose model struggle with a question about a document it’s never seen, even if the document is short and simple?

In the first code cell, try a question using completely different wording than any document (e.g. asking about the office stipend using none of the same words). What happens to the scores, and why?

If retrieval picks the wrong document, can the generate step “notice” and correct it? Why or why not, based on what you saw in the “what generate actually receives” section?

Suppose you swapped these three documents for a completely different collection, recipes instead of HR policies. Would word-matching’s weakness still be a problem? Give a concrete example of two words in that new domain that mean the same thing but share no letters.

You can now: explain what retrieval and generation each do, and you’ve seen exactly where word-matching retrieval breaks down.

But one problem remains: “holiday” and “vacation” mean the same thing to a person but share zero letters to this retriever. The next chapter fixes that by comparing meaning instead of spelling.

Next → Chapter 2: What Are Embeddings?