Chapter 2: What Are Embeddings? - Velmurugan Narayanasamy

Chapter 1 ended on a specific failure. Our retrieval system counted shared words between a question and a document, and it worked, right up until someone asked “do unused holidays roll over to next year?” instead of using the word “vacation.” Same question, different words, zero words in common with the document that actually answers it. The retriever picked the wrong document, or a tie, and the generate step never had a chance to save it.

“Holiday” and “vacation” mean the same thing to a person. To a computer comparing spelling, they’re as unrelated as “holiday” and “giraffe.” This chapter fixes that, by changing what we compare: not words, but meaning.

Chapter 2 of 10 · Foundations

Where we left off: Chapter 1’s word-matching retriever tied or lost on a question that used “holidays” instead of “vacation,” because it only compares spelling.

What we fix now: Turn text into vectors so meaning, not spelling, determines what counts as similar, built from scratch with nothing but word co-occurrence counts and a bit of linear algebra.

By the end, you can:

explain what an embedding is and what cosine similarity measures
build a tiny embedding model from scratch and use it to fix Chapter 1’s exact failure
name two concrete limits of this toy model, and why production models handle them differently

The core idea¶

An embedding turns a piece of text into a list of numbers, a vector, chosen so that texts with similar meaning end up as vectors that are close together, and texts with different meaning end up far apart. “Vacation” and “holiday” should land near each other in this space. “Vacation” and “printer” should not.

That’s the entire idea. Once text is a vector, “how similar are these two pieces of text” becomes a question with actual math behind it: measure the distance, or more commonly the angle, between two vectors. Close together (small angle) means similar meaning. Far apart (large angle) means unrelated.

The hard part is earning that property in the first place: how do you turn a sentence into numbers such that “meaning” falls out of the geometry? That’s what the rest of this chapter builds, from scratch, so you can see exactly where those numbers come from instead of treating them as a black box.

Where real embedding numbers come from¶

Production systems (OpenAI’s, Cohere’s, the open-source sentence-transformers models) train large neural networks on billions of words to produce these vectors. We can’t run that in a browser tab. But the core mechanism those models build on is much older and simple enough to build live, right here, with nothing but counting and a bit of linear algebra:

Take a body of text (a corpus).
For every pair of words, count how often they show up near each other across that corpus. This gives a big table: one row per word, and each row is a fingerprint of “which words tend to appear around me.”
Two words that keep similar company end up with similar fingerprints, even if the two words themselves are never adjacent. “Holiday” and “vacation” both tend to show up near words like “days,” “unused,” and “employees,” so their fingerprints look alike, well before anything is told what either word “means.”
Compress those long, mostly-redundant fingerprints down into a short list of numbers per word (a handful of dimensions instead of hundreds), keeping whatever pattern explains the most variation. That compressed fingerprint is the embedding.

This co-occurrence-then-compress approach is the same lineage as classic techniques like LSA, and conceptually close to how GloVe (a well-known pretrained embedding model) was built, just at a vastly smaller scale here: dozens of sentences instead of billions of words. To be clear about what you’re about to run: this is not a download of some official pretrained file. It’s a tiny embedding model, trained live, in your browser, on the twenty sentences below, in a fraction of a second.

import numpy as np
import re

# A small corpus, just enough to give our target words real context.
# Notice "vacation" and "holiday" keep showing up near the same
# neighboring words, even though they never sit right next to each
# other in any sentence.
corpus = [
    "Employees may carry over unused vacation days into the next calendar year.",
    "Staff can roll over unused holiday days into next year.",
    "Workers get paid time off and vacation days every year.",
    "Most employees take a vacation or holiday once a year.",
    "Unused vacation time can roll over to next year for many employees.",
    "Holiday leave and vacation leave are treated the same way at most companies.",
    "Some employees prefer to use their holiday days instead of saving them.",
    "Vacation days and holiday days both count as paid time off.",
    "Expense reports must be submitted within days of the purchase.",
    "Employees submit an expense report after every business purchase.",
    "The finance team reviews each expense report and each purchase receipt.",
    "A purchase over the limit needs approval before the expense is reimbursed.",
    "Expense claims and purchase receipts are reviewed together by finance.",
    "Remote employees receive a home office stipend to set up their workspace.",
    "The company gives remote workers a stipend for home office equipment.",
    "Employees working remote from home can request an office stipend.",
    "A stipend helps remote staff pay for office furniture at home.",
    "The office stipend is a one time payment for remote employees.",
    "Vacation policy, expense policy, and remote work policy are all in the handbook.",
    "Every employee should read the handbook for vacation and expense rules.",
]

def tokenize(text):
    return re.findall(r"[a-z]+", text.lower())

tokens_per_sentence = [tokenize(s) for s in corpus]
vocab = sorted(set(w for sent in tokens_per_sentence for w in sent))
word_index = {w: i for i, w in enumerate(vocab)}
V = len(vocab)

# Co-occurrence: how often does each word appear within 4 words of
# each other word, counted across the whole corpus.
window = 4
cooc = np.zeros((V, V))
for sent in tokens_per_sentence:
    for i, w in enumerate(sent):
        for j in range(max(0, i - window), min(len(sent), i + window + 1)):
            if i != j:
                cooc[word_index[w], word_index[sent[j]]] += 1

# Compress: log-smooth the counts, then use SVD to squeeze the big
# co-occurrence table down to a small number of dense dimensions.
X = np.log1p(cooc)
U, S, Vt = np.linalg.svd(X, full_matrices=False)

DIM = 20
word_vectors = U[:, :DIM] * S[:DIM]

def embedding(word):
    return word_vectors[word_index[word]] if word in word_index else None

print(f"Vocabulary size: {V} words")
print(f"Each word is now a vector of {DIM} numbers.")
print("First 5 numbers for 'vacation':", embedding("vacation")[:5].round(2))

Measuring “close together”¶

With every word turned into a vector, we need a way to measure how close two vectors are. The standard tool is cosine similarity: the cosine of the angle between two vectors.

\text{cosine\_similarity}(a, b) = \frac{a \cdot b}{\lVert a \rVert \, \lVert b \rVert}

(1)

where $a \cdot b$ is the dot product ( $\sum_i a_i b_i$ ) and $\lVert a \rVert$ is the vector’s length ( $\sqrt{\sum_i a_i^2}$ ). The result always falls between -1 and 1: values closer to 1 indicate stronger directional similarity, values near 0 indicate weak directional similarity, and negative values indicate vectors oriented in opposing directions. Geometrically that’s precise; whether “opposite direction” also means “opposite meaning” is a separate question, and the answer depends on the specific embedding model, not something to assume automatically. Dividing by the lengths is what makes this about direction (meaning) rather than magnitude (how long the vector happens to be), which matters because nothing here guarantees every word’s vector comes out the same length.

You don’t need to memorize this formula to use embeddings, most libraries compute it for you in one line, but it’s worth seeing once so “similarity score” stops being magic.

def cosine_similarity(a, b):
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

pairs = [
    ("holiday", "vacation"),
    ("holiday", "expense"),
    ("holiday", "stipend"),
    ("vacation", "stipend"),
    ("expense", "purchase"),
    ("remote", "office"),
]

for w1, w2 in pairs:
    score = cosine_similarity(embedding(w1), embedding(w2))
    print(f"{w1:>10} vs {w2:<10} {score:.3f}")

“holiday” and “vacation” score meaningfully higher than “holiday” paired with “expense” or “stipend”, topic words that never showed up in the same sentences. Nobody told this model that holidays and vacations are the same idea; it fell out of which words tend to surround them in the corpus. Try editing the pairs list above with your own word combinations from the corpus and re-run the cell.

def embed_text(text):
    words = [w for w in tokenize(text) if w in word_index]
    if not words:
        return np.zeros(DIM)
    return np.mean([embedding(w) for w in words], axis=0)

documents = [
    "Employees may carry over up to 5 unused vacation days into the next calendar year.",
    "Expense reports must be submitted within 30 days of the purchase date.",
    "Remote employees are provided a one-time home office stipend of 500 euros.",
]

# The exact question that broke word-matching retrieval in Chapter 1.
question = "do unused holidays roll over to next year?"

question_vector = embed_text(question)
ranked = sorted(
    documents,
    key=lambda d: cosine_similarity(question_vector, embed_text(d)),
    reverse=True,
)

for d in ranked:
    score = cosine_similarity(question_vector, embed_text(d))
    print(f"{score:.3f}  {d}")

In Chapter 1, this exact question tied or lost against the wrong document, because it shares zero words with the vacation policy. Here, the vacation document wins clearly, not because it shares more words (it still doesn’t), but because “holidays roll over to next year” and “vacation days into the next calendar year” point in nearly the same direction in this vector space. Edit question above and try your own phrasing, including ones that share no words at all with any document, and re-run the cell.

Before · word matching (Chapter 1)

Same question: “do unused holidays roll over to next year?”

Shared words with the vacation document: 4, but all four are generic connectors (“unused,” “over,” “next,” “to”). Zero overlap on the words that actually carry the topic, “holiday” or “vacation.”

After · embeddings (this chapter)

Same question: “do unused holidays roll over to next year?”

Cosine similarity with the vacation document: 0.943, next closest document only 0.515. This win comes from genuine semantic closeness between “holiday” and “vacation,” not lucky word overlap.

Seeing it, not just measuring it¶

Numbers on their own can be hard to trust. Here’s the same idea as a picture: compress each word down to just 2 dimensions (instead of 20) and plot them. Words with related meaning should end up near each other on the page.

import matplotlib.pyplot as plt

words_to_plot = [
    "vacation", "holiday", "leave",
    "expense", "purchase", "receipt",
    "remote", "stipend", "office", "employees",
]

vectors_2d = U[:, :2] * S[:2]

fig, ax = plt.subplots(figsize=(6, 5))
for w in words_to_plot:
    x, y = vectors_2d[word_index[w]]
    ax.scatter(x, y)
    ax.annotate(w, (x, y), textcoords="offset points", xytext=(5, 5))
ax.set_title("Words compressed to 2 dimensions")
plt.show()

Look for the two loose clusters: vacation-and-holiday-flavored words on one side, remote-and-office-flavored words on another, with expense-and-purchase words forming their own neighborhood. Nothing forced that grouping; it’s a side effect of which words share sentence neighborhoods in the corpus above.

Where it breaks

This tiny model has real limits, worth naming honestly:

It only knows words it has seen. Ask about “PTO” or “furlough,” words that never appeared in our twenty sentences, and there’s no vector for them at all: embedding("furlough") returns nothing to work with. Real production models are trained on billions of words and handle this far better (though never perfectly), and some can even build reasonable vectors for words they’ve never seen, by breaking them into familiar sub-word pieces.
A word gets exactly one vector, always. “Remote” means the same thing whether the sentence is about remote work or a TV remote; our model can’t tell those apart, because it never looks at context, just co-occurrence statistics baked in ahead of time. The neural models mentioned above generate a different vector for the same word depending on the sentence it’s actually in, which is a large part of why they perform so much better in practice.
This doesn’t scale by counting. Comparing a question against three documents is instant. Comparing it against three million documents, one cosine similarity at a time, is not, and that’s a real problem once a RAG system has a real-sized document collection behind it.

If you want to see a real, production-grade embedding model in action (the kind that solves the two bullet points above), there’s a bonus cell below using an actual pretrained transformer model. It won’t run in this page, the same runnable-but-not-editable limitation from Chapter 1 applies, and this one additionally needs a full Python environment (pip install, a real neural network) that a browser tab can’t provide. Open it in Colab to actually run it:

Bonus (Colab only): a real pretrained embedding model¶

This cell installs sentence-transformers and runs all-MiniLM-L6-v2, a real pretrained transformer model trained on far more than twenty sentences. It’s here so you can compare its output to the toy model above, not because you need to understand how it works internally yet, that’s a later chapter.

# Colab only: this needs torch + a real download, which won't run
# in this page's in-browser Python (no torch in WASM). Open this
# notebook in Colab (badge above) to run this cell.
!pip install -q sentence-transformers

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("all-MiniLM-L6-v2")

documents = [
    "Employees may carry over up to 5 unused vacation days into the next calendar year.",
    "Expense reports must be submitted within 30 days of the purchase date.",
    "Remote employees are provided a one-time home office stipend of 500 euros.",
]
question = "do unused holidays roll over to next year?"

doc_embeddings = model.encode(documents, convert_to_tensor=True)
question_embedding = model.encode(question, convert_to_tensor=True)

scores = util.cos_sim(question_embedding, doc_embeddings)[0]
for score, doc in sorted(zip(scores, documents), reverse=True):
    print(f"{float(score):.3f}  {doc}")

Quick check¶

Before moving on, answer for yourself (no peeking ahead):

In your own words, what does it mean for two words to be “close together” as vectors, and what does the model actually measure to decide that, since nobody tells it what any word means?

The toy model in this chapter never saw the word “furlough.” What happens if you try to embed a question containing it, and why?

Why does a single fixed vector per word struggle with a word that has more than one meaning, such as “remote”? What would need to change to fix that?

Comparing a question against 3 documents with cosine similarity is instant. What do you think starts to break down once there are 3 million documents instead of 3, and why might “compare against everything, one at a time” stop being a workable strategy?

Chapter 1’s retriever technically got the vacation-vs-holiday question right too, just for the wrong reason. Why does “got the right document, for the wrong reason” matter, and can you think of a slightly different question where that same lucky word overlap would have picked the wrong document instead?

You can now: turn text into vectors and use cosine similarity to compare meaning instead of spelling, and you’ve watched it genuinely fix Chapter 1’s exact failure.

But one problem remains: every document so far has been one sentence. Real documents are pages long, and averaging an entire multi-topic document into one vector blends everything together.

Next → Chapter 3: Chunking Long Documents