Chapter 2 told one side of a story: word-overlap retrieval lost to embeddings on a question about “holidays” that shared zero words with the document that answered it. Meaning beat spelling. This chapter tells the other side: there are questions where embeddings lose outright to plain keyword matching, and the reason is just as mechanical as Chapter 2’s was.
Where we left off: Chapter 2 told one side of a story: word matching lost to embeddings on a question about “holidays” that shared zero words with the right document.
What we fix now: See the other side: a question where embeddings lose outright to plain keyword matching, then combine both methods with Reciprocal Rank Fusion.
By the end, you can:
explain why a tokenizer that drops digits makes embeddings blind to codes and numbers
build BM25 from scratch and combine it with embeddings using Reciprocal Rank Fusion
identify a real failure mode of RRF itself: a meaningless tied score still gets a real rank
Source
import numpy as np
import re
import math
# Same technique as Chapters 2-6, extended with one more section and
# a bit more vocabulary for it.
training_corpus = [
"Employees may carry over unused vacation days into the next calendar year.",
"Staff can roll over unused holiday days into next year.",
"Workers get paid time off and vacation days every year.",
"Most employees take a vacation or holiday once a year.",
"Unused vacation time can roll over to next year for many employees.",
"Holiday leave and vacation leave are treated the same way at most companies.",
"Some employees prefer to use their holiday days instead of saving them.",
"Vacation days and holiday days both count as paid time off.",
"Expense reports must be submitted within days of the purchase.",
"Employees submit an expense report after every business purchase.",
"The finance team reviews each expense report and each purchase receipt.",
"A purchase over the limit needs approval before the expense is reimbursed.",
"Expense claims and purchase receipts are reviewed together by finance.",
"Remote employees receive a home office stipend to set up their workspace.",
"The company gives remote workers a stipend for home office equipment.",
"Employees working remote from home can request an office stipend.",
"A stipend helps remote staff pay for office furniture at home.",
"The office stipend is a one time payment for remote employees.",
"Vacation policy, expense policy, and remote work policy are all in the handbook.",
"Every employee should read the handbook for vacation and expense rules.",
"Employees who are sick should notify their manager as soon as possible.",
"Sick leave is paid time off for employees who are too unwell to work.",
"A doctor's note may be required after three consecutive sick days.",
"Unused sick leave does not roll over into the next year.",
"Employees should stay home and rest when they are sick.",
"Standard working hours are nine to five, Monday through Friday.",
"Employees may request a flexible schedule with manager approval.",
"Overtime must be approved in advance by a manager.",
"Most staff work standard business hours unless a flexible schedule is approved.",
"Employees are expected to treat colleagues and customers with respect.",
"Harassment or discrimination of any kind will not be tolerated.",
"Violations of the code of conduct may result in disciplinary action.",
"Every employee must follow the code of conduct and treat others with respect.",
"Company laptops must be encrypted and password protected.",
"Employees should not install unauthorized software on company devices.",
"Report a lost or stolen laptop to IT immediately.",
"IT equipment must be returned when an employee leaves the company.",
"The IT team manages laptops, software, and device security for the company.",
"Contact the helpdesk if you have a technical problem with your computer.",
"The support team can help reset passwords and fix login issues.",
"Call the helpdesk for any urgent technical problem during business hours.",
]
def tokenize(text):
# Letters only: this is the same tokenizer Chapters 2-6 used for
# embeddings, and it quietly drops every digit.
return re.findall(r"[a-z']+", text.lower())
tokens_per_sentence = [tokenize(s) for s in training_corpus]
vocab = sorted(set(w for sent in tokens_per_sentence for w in sent))
word_index = {w: i for i, w in enumerate(vocab)}
V = len(vocab)
window = 4
cooc = np.zeros((V, V))
for sent in tokens_per_sentence:
for i, w in enumerate(sent):
for j in range(max(0, i - window), min(len(sent), i + window + 1)):
if i != j:
cooc[word_index[w], word_index[sent[j]]] += 1
X = np.log1p(cooc)
U, S, Vt = np.linalg.svd(X, full_matrices=False)
DIM = 25
word_vectors = U[:, :DIM] * S[:DIM]
def embedding(word):
return word_vectors[word_index[word]] if word in word_index else None
def embed_text(text):
words = [w for w in tokenize(text) if w in word_index]
if not words:
return np.zeros(DIM)
return np.mean([embedding(w) for w in words], axis=0)
def cosine_similarity(a, b):
norm = np.linalg.norm(a) * np.linalg.norm(b)
return float(np.dot(a, b) / norm) if norm else 0.0
handbook_document = """Vacation Policy
Employees may carry over up to 5 unused vacation days into the next calendar year. Time off should be requested from a manager at least two weeks in advance when possible.
Sick Leave Policy
Employees who are too unwell to work should notify their manager as soon as possible. Sick leave is paid time off and does not roll over into the next year. A doctor\'s note may be required after three consecutive sick days.
Expense Policy
Expense reports must be submitted within 30 days of the purchase date. The finance team reviews each expense report and matching purchase receipt before reimbursement.
Remote Work Policy
Remote employees are provided a one-time home office stipend of 500 euros. The stipend covers office furniture and basic equipment needed to work from home.
Working Hours
Standard working hours are nine to five, Monday through Friday. Employees may request a flexible schedule with manager approval, and any overtime must be approved in advance.
Code of Conduct
Employees are expected to treat colleagues and customers with respect. Harassment or discrimination of any kind will not be tolerated, and violations may result in disciplinary action.
IT Support
For urgent technical problems, contact the helpdesk at extension 4477. For non-urgent requests, submit a ticket through the internal support portal."""
section_chunks = [p.strip() for p in handbook_document.split("\n\n")]
print(f"Ready: {len(section_chunks)} sections (added IT Support), {V} words in vocabulary.")question = "what department is 4477"
ranked = sorted(
section_chunks,
key=lambda c: cosine_similarity(embed_text(question), embed_text(c)),
reverse=True,
)
for c in ranked[:3]:
score = cosine_similarity(embed_text(question), embed_text(c))
print(f"{score:.3f} {c.splitlines()[0]}")Embeddings confidently return the wrong section. The reason is mechanical, not mysterious: the tokenizer used for embeddings keeps letters only, so “4477” disappears from the question entirely before embedding even starts, and “extension” was never one of the roughly 200 words the toy embedding model was trained on in the first place. What’s left, “what,” “department,” “is,” “for,” carries almost no real signal, so whatever the embedding model returns is closer to a guess than a retrieval.
Same question: “what department is 4477”
Top result: Remote Work Policy at 0.560, the wrong section, and IT Support doesn’t even appear in the top 3. “4477” vanished before embedding even started, letters-only tokenization drops it entirely.
Same question: “what department is 4477”
Top result: IT Support at 1.860, a wide margin over everything else. BM25 keeps digits, so “4477” survives as an exact, distinctive token that only appears in one section.
Real keyword search: BM25¶
Chapter 1 used a naive word-overlap count. Real keyword search systems use something more careful: BM25, which scores a document higher when it contains a query term more often (with diminishing returns, the fourth mention doesn’t matter as much as the first), adjusted for how long the document is, and weighted by how rare that term is across the whole collection (a common word like “the” barely counts; a rare word like “helpdesk” counts for a lot). It’s simple enough to build from scratch and still genuinely used in production search today, often as one half of a hybrid system.
def bm25_tokenize(text):
# Keeps digits, unlike the embedding tokenizer, so "4477" survives
# as a real token here.
return re.findall(r"[a-z0-9']+", text.lower())
class BM25:
def __init__(self, documents, k1=1.5, b=0.75):
self.k1 = k1
self.b = b
self.doc_tokens = [bm25_tokenize(d) for d in documents]
self.doc_lens = [len(toks) for toks in self.doc_tokens]
self.avgdl = sum(self.doc_lens) / len(self.doc_lens)
self.N = len(documents)
doc_freq = {}
for toks in self.doc_tokens:
for term in set(toks):
doc_freq[term] = doc_freq.get(term, 0) + 1
# Rare terms get a high idf (count for a lot); common terms
# get a low one (barely count).
self.idf = {
term: math.log((self.N - n + 0.5) / (n + 0.5) + 1)
for term, n in doc_freq.items()
}
def score(self, query, doc_index):
term_freq = {}
for t in self.doc_tokens[doc_index]:
term_freq[t] = term_freq.get(t, 0) + 1
dl = self.doc_lens[doc_index]
total = 0.0
for term in bm25_tokenize(query):
if term not in self.idf:
continue
f = term_freq.get(term, 0)
idf = self.idf[term]
total += idf * (f * (self.k1 + 1)) / (
f + self.k1 * (1 - self.b + self.b * dl / self.avgdl)
)
return total
bm25 = BM25(section_chunks)
ranked_bm25 = sorted(
range(len(section_chunks)),
key=lambda i: bm25.score(question, i),
reverse=True,
)
for i in ranked_bm25[:3]:
print(f"{bm25.score(question, i):.3f} {section_chunks[i].splitlines()[0]}")BM25 finds it immediately, and by a wide margin: “4477” and “extension” are exact, distinctive tokens that only appear in one section, exactly the kind of signal keyword search is built for and embeddings, in this toy model, can’t represent at all.
Combining both: Reciprocal Rank Fusion¶
Neither method should simply replace the other, Chapter 2’s “holidays”
question still needs embeddings, this chapter’s “4477” question needs
BM25. The two scores can’t just be added together directly: cosine
similarity lives between -1 and 1, BM25 scores are unbounded and on a
completely different scale. Reciprocal Rank Fusion (RRF) sidesteps
that entirely by ignoring raw scores and combining rank positions
instead: a document gets 1 / (k + rank) points from each method it
was retrieved by, added together across methods.
def rrf_combine(question, chunks, bm25_index, k=60):
embed_ranked = sorted(
range(len(chunks)),
key=lambda i: cosine_similarity(embed_text(question), embed_text(chunks[i])),
reverse=True,
)
bm25_ranked = sorted(
range(len(chunks)),
key=lambda i: bm25_index.score(question, i),
reverse=True,
)
rrf_scores = {}
for rank, i in enumerate(embed_ranked, start=1):
rrf_scores[i] = rrf_scores.get(i, 0) + 1 / (k + rank)
for rank, i in enumerate(bm25_ranked, start=1):
rrf_scores[i] = rrf_scores.get(i, 0) + 1 / (k + rank)
return sorted(rrf_scores.items(), key=lambda kv: kv[1], reverse=True)
combined = rrf_combine(question, section_chunks, bm25)
for idx, score in combined[:3]:
print(f"{score:.4f} {section_chunks[idx].splitlines()[0]}")Hybrid search gets it right. Now the other direction, to make sure nothing regressed: Chapter 2’s original synonym question, the one embeddings needed to solve in the first place.
holidays_question = "do unused holidays roll over to next year?"
combined_holidays = rrf_combine(holidays_question, section_chunks, bm25)
for idx, score in combined_holidays[:3]:
print(f"{score:.4f} {section_chunks[idx].splitlines()[0]}")Still correct. Hybrid search didn’t have to choose between Chapter 2’s win and this chapter’s win, it keeps both, because each method still contributes its own ranking to the fusion regardless of what the other one thinks.
bare_query = "4477"
print("Embedding scores (all tied at 0.0, meaningless):")
for c in section_chunks:
print(f" {cosine_similarity(embed_text(bare_query), embed_text(c)):.3f} {c.splitlines()[0]}")
print()
combined_bare = rrf_combine(bare_query, section_chunks, bm25)
print("RRF winner:", section_chunks[combined_bare[0][0]].splitlines()[0], "(likely wrong)")my_question = "who handles password reset issues?"
my_k = 60
combined = rrf_combine(my_question, section_chunks, bm25, k=my_k)
for idx, score in combined[:3]:
print(f"{score:.4f} {section_chunks[idx].splitlines()[0]}")Quick check¶
Before moving on, answer for yourself (no peeking ahead):
Why does the embedding tokenizer losing digits matter so much more for the “4477” question than it would for a typical question about vacation or sick leave?
Check your reasoning ▸
For a typical question, plenty of other words (“vacation,” “sick,” “carry over”) still carry real signal even if a stray number gets dropped. For the “4477” question, the number is essentially the entire meaningful content: once it’s stripped out, almost nothing distinctive is left for the embedding model to work with.
Why can’t cosine similarity scores and BM25 scores just be added together directly? What does RRF compare instead, and why does that sidestep the problem?
Check your reasoning ▸
Cosine similarity is bounded between -1 and 1, while BM25 scores are unbounded and on a completely different scale, so adding them directly would let whichever score happens to have bigger numbers dominate for no principled reason. RRF sidesteps this by ignoring raw scores entirely and combining rank positions instead, which sit on the same scale (1st, 2nd, 3rd) no matter which method produced them.
The bare “4477” query broke RRF even though BM25 alone got it right. What specifically went wrong, and is that BM25’s fault, the embedding model’s fault, or RRF’s fault?
Check your reasoning ▸
A strong answer traces the mechanism precisely: the embedding side produced a meaningless, tied 0.0 score across every section, but RRF still assigns that meaningless ranking a real rank position, and that arbitrary rank ends up outweighing BM25’s genuinely confident, correctly ranked result. It’s really a gap in RRF’s assumption that every method’s ranking carries some signal, not a bug in either individual method.
If you were building a real system and knew your users mostly asked natural-language questions with the occasional order number or product code mixed in, would you lean more on tuning
k, improving the embedding model’s vocabulary, or something else entirely?
Check your reasoning ▸
There’s no single right answer here, but a strong response reasons about trade-offs: tuning k reweights an existing symptom without fixing the cause, improving vocabulary helps but numbers and codes are an open-ended set no fixed vocabulary fully covers, and “something else” (like detecting when a query is mostly non-word tokens and leaning harder on BM25 for it) targets the actual mechanism instead of working around it.
Chapter 2 showed embeddings winning where word-matching failed (“holidays” vs “vacation”). This chapter shows the opposite: embeddings losing where keyword matching wins. Does this contradict Chapter 2, or is something else going on?
Check your reasoning ▸
It doesn’t contradict Chapter 2, it completes the picture. Chapter 2’s example needed meaning, two different spellings, one idea, while this chapter’s example needs an exact token match, a number that only ever means one specific thing. Both are real, and a strong answer recognizes there’s no single method that’s simply “better,” which is exactly why hybrid search combining both exists.
You can now: combine embeddings and keyword search with Reciprocal Rank Fusion, and you’ve seen a real case where combining them still breaks: a query with no usable signal from one side poisoning the fused ranking.
But one problem remains: every retrieval method in this series so far has worked the same way: embed the question, embed each candidate independently, compare the two fixed vectors after the fact. That misses signal a closer look could catch.
Next → Chapter 8: Re-ranking