Chapter 7: Hybrid Search - Velmurugan Narayanasamy

Chapter 2 told one side of a story: word-overlap retrieval lost to embeddings on a question about “holidays” that shared zero words with the document that answered it. Meaning beat spelling. This chapter tells the other side: there are questions where embeddings lose outright to plain keyword matching, and the reason is just as mechanical as Chapter 2’s was.

Chapter 7 of 10 · Better retrieval

Where we left off: Chapter 2 told one side of a story: word matching lost to embeddings on a question about “holidays” that shared zero words with the right document.

What we fix now: See the other side: a question where embeddings lose outright to plain keyword matching, then combine both methods with Reciprocal Rank Fusion.

By the end, you can:

explain why a tokenizer that drops digits makes embeddings blind to codes and numbers
build BM25 from scratch and combine it with embeddings using Reciprocal Rank Fusion
identify a real failure mode of RRF itself: a meaningless tied score still gets a real rank

import numpy as np
import re
import math

# Same technique as Chapters 2-6, extended with one more section and
# a bit more vocabulary for it.
training_corpus = [
    "Employees may carry over unused vacation days into the next calendar year.",
    "Staff can roll over unused holiday days into next year.",
    "Workers get paid time off and vacation days every year.",
    "Most employees take a vacation or holiday once a year.",
    "Unused vacation time can roll over to next year for many employees.",
    "Holiday leave and vacation leave are treated the same way at most companies.",
    "Some employees prefer to use their holiday days instead of saving them.",
    "Vacation days and holiday days both count as paid time off.",
    "Expense reports must be submitted within days of the purchase.",
    "Employees submit an expense report after every business purchase.",
    "The finance team reviews each expense report and each purchase receipt.",
    "A purchase over the limit needs approval before the expense is reimbursed.",
    "Expense claims and purchase receipts are reviewed together by finance.",
    "Remote employees receive a home office stipend to set up their workspace.",
    "The company gives remote workers a stipend for home office equipment.",
    "Employees working remote from home can request an office stipend.",
    "A stipend helps remote staff pay for office furniture at home.",
    "The office stipend is a one time payment for remote employees.",
    "Vacation policy, expense policy, and remote work policy are all in the handbook.",
    "Every employee should read the handbook for vacation and expense rules.",
    "Employees who are sick should notify their manager as soon as possible.",
    "Sick leave is paid time off for employees who are too unwell to work.",
    "A doctor's note may be required after three consecutive sick days.",
    "Unused sick leave does not roll over into the next year.",
    "Employees should stay home and rest when they are sick.",
    "Standard working hours are nine to five, Monday through Friday.",
    "Employees may request a flexible schedule with manager approval.",
    "Overtime must be approved in advance by a manager.",
    "Most staff work standard business hours unless a flexible schedule is approved.",
    "Employees are expected to treat colleagues and customers with respect.",
    "Harassment or discrimination of any kind will not be tolerated.",
    "Violations of the code of conduct may result in disciplinary action.",
    "Every employee must follow the code of conduct and treat others with respect.",
    "Company laptops must be encrypted and password protected.",
    "Employees should not install unauthorized software on company devices.",
    "Report a lost or stolen laptop to IT immediately.",
    "IT equipment must be returned when an employee leaves the company.",
    "The IT team manages laptops, software, and device security for the company.",
    "Contact the helpdesk if you have a technical problem with your computer.",
    "The support team can help reset passwords and fix login issues.",
    "Call the helpdesk for any urgent technical problem during business hours.",
]

def tokenize(text):
    # Letters only: this is the same tokenizer Chapters 2-6 used for
    # embeddings, and it quietly drops every digit.
    return re.findall(r"[a-z']+", text.lower())

tokens_per_sentence = [tokenize(s) for s in training_corpus]
vocab = sorted(set(w for sent in tokens_per_sentence for w in sent))
word_index = {w: i for i, w in enumerate(vocab)}
V = len(vocab)
window = 4
cooc = np.zeros((V, V))
for sent in tokens_per_sentence:
    for i, w in enumerate(sent):
        for j in range(max(0, i - window), min(len(sent), i + window + 1)):
            if i != j:
                cooc[word_index[w], word_index[sent[j]]] += 1
X = np.log1p(cooc)
U, S, Vt = np.linalg.svd(X, full_matrices=False)
DIM = 25
word_vectors = U[:, :DIM] * S[:DIM]

def embedding(word):
    return word_vectors[word_index[word]] if word in word_index else None

def embed_text(text):
    words = [w for w in tokenize(text) if w in word_index]
    if not words:
        return np.zeros(DIM)
    return np.mean([embedding(w) for w in words], axis=0)

def cosine_similarity(a, b):
    norm = np.linalg.norm(a) * np.linalg.norm(b)
    return float(np.dot(a, b) / norm) if norm else 0.0

handbook_document = """Vacation Policy
Employees may carry over up to 5 unused vacation days into the next calendar year. Time off should be requested from a manager at least two weeks in advance when possible.

Sick Leave Policy
Employees who are too unwell to work should notify their manager as soon as possible. Sick leave is paid time off and does not roll over into the next year. A doctor\'s note may be required after three consecutive sick days.

Expense Policy
Expense reports must be submitted within 30 days of the purchase date. The finance team reviews each expense report and matching purchase receipt before reimbursement.

Remote Work Policy
Remote employees are provided a one-time home office stipend of 500 euros. The stipend covers office furniture and basic equipment needed to work from home.

Working Hours
Standard working hours are nine to five, Monday through Friday. Employees may request a flexible schedule with manager approval, and any overtime must be approved in advance.

Code of Conduct
Employees are expected to treat colleagues and customers with respect. Harassment or discrimination of any kind will not be tolerated, and violations may result in disciplinary action.

IT Support
For urgent technical problems, contact the helpdesk at extension 4477. For non-urgent requests, submit a ticket through the internal support portal."""

section_chunks = [p.strip() for p in handbook_document.split("\n\n")]
print(f"Ready: {len(section_chunks)} sections (added IT Support), {V} words in vocabulary.")

question = "what department is 4477"

ranked = sorted(
    section_chunks,
    key=lambda c: cosine_similarity(embed_text(question), embed_text(c)),
    reverse=True,
)
for c in ranked[:3]:
    score = cosine_similarity(embed_text(question), embed_text(c))
    print(f"{score:.3f}  {c.splitlines()[0]}")

Embeddings confidently return the wrong section. The reason is mechanical, not mysterious: the tokenizer used for embeddings keeps letters only, so “4477” disappears from the question entirely before embedding even starts, and “extension” was never one of the roughly 200 words the toy embedding model was trained on in the first place. What’s left, “what,” “department,” “is,” “for,” carries almost no real signal, so whatever the embedding model returns is closer to a guess than a retrieval.

Before · embeddings alone

Same question: “what department is 4477”

Top result: Remote Work Policy at 0.560, the wrong section, and IT Support doesn’t even appear in the top 3. “4477” vanished before embedding even started, letters-only tokenization drops it entirely.

After · BM25 keyword search

Same question: “what department is 4477”

Top result: IT Support at 1.860, a wide margin over everything else. BM25 keeps digits, so “4477” survives as an exact, distinctive token that only appears in one section.

Real keyword search: BM25¶

Chapter 1 used a naive word-overlap count. Real keyword search systems use something more careful: BM25, which scores a document higher when it contains a query term more often (with diminishing returns, the fourth mention doesn’t matter as much as the first), adjusted for how long the document is, and weighted by how rare that term is across the whole collection (a common word like “the” barely counts; a rare word like “helpdesk” counts for a lot). It’s simple enough to build from scratch and still genuinely used in production search today, often as one half of a hybrid system.

def bm25_tokenize(text):
    # Keeps digits, unlike the embedding tokenizer, so "4477" survives
    # as a real token here.
    return re.findall(r"[a-z0-9']+", text.lower())

class BM25:
    def __init__(self, documents, k1=1.5, b=0.75):
        self.k1 = k1
        self.b = b
        self.doc_tokens = [bm25_tokenize(d) for d in documents]
        self.doc_lens = [len(toks) for toks in self.doc_tokens]
        self.avgdl = sum(self.doc_lens) / len(self.doc_lens)
        self.N = len(documents)

        doc_freq = {}
        for toks in self.doc_tokens:
            for term in set(toks):
                doc_freq[term] = doc_freq.get(term, 0) + 1
        # Rare terms get a high idf (count for a lot); common terms
        # get a low one (barely count).
        self.idf = {
            term: math.log((self.N - n + 0.5) / (n + 0.5) + 1)
            for term, n in doc_freq.items()
        }

    def score(self, query, doc_index):
        term_freq = {}
        for t in self.doc_tokens[doc_index]:
            term_freq[t] = term_freq.get(t, 0) + 1
        dl = self.doc_lens[doc_index]

        total = 0.0
        for term in bm25_tokenize(query):
            if term not in self.idf:
                continue
            f = term_freq.get(term, 0)
            idf = self.idf[term]
            total += idf * (f * (self.k1 + 1)) / (
                f + self.k1 * (1 - self.b + self.b * dl / self.avgdl)
            )
        return total

bm25 = BM25(section_chunks)

ranked_bm25 = sorted(
    range(len(section_chunks)),
    key=lambda i: bm25.score(question, i),
    reverse=True,
)
for i in ranked_bm25[:3]:
    print(f"{bm25.score(question, i):.3f}  {section_chunks[i].splitlines()[0]}")

BM25 finds it immediately, and by a wide margin: “4477” and “extension” are exact, distinctive tokens that only appear in one section, exactly the kind of signal keyword search is built for and embeddings, in this toy model, can’t represent at all.

Combining both: Reciprocal Rank Fusion¶

Neither method should simply replace the other, Chapter 2’s “holidays” question still needs embeddings, this chapter’s “4477” question needs BM25. The two scores can’t just be added together directly: cosine similarity lives between -1 and 1, BM25 scores are unbounded and on a completely different scale. Reciprocal Rank Fusion (RRF) sidesteps that entirely by ignoring raw scores and combining rank positions instead: a document gets 1 / (k + rank) points from each method it was retrieved by, added together across methods.

def rrf_combine(question, chunks, bm25_index, k=60):
    embed_ranked = sorted(
        range(len(chunks)),
        key=lambda i: cosine_similarity(embed_text(question), embed_text(chunks[i])),
        reverse=True,
    )
    bm25_ranked = sorted(
        range(len(chunks)),
        key=lambda i: bm25_index.score(question, i),
        reverse=True,
    )

    rrf_scores = {}
    for rank, i in enumerate(embed_ranked, start=1):
        rrf_scores[i] = rrf_scores.get(i, 0) + 1 / (k + rank)
    for rank, i in enumerate(bm25_ranked, start=1):
        rrf_scores[i] = rrf_scores.get(i, 0) + 1 / (k + rank)

    return sorted(rrf_scores.items(), key=lambda kv: kv[1], reverse=True)

combined = rrf_combine(question, section_chunks, bm25)
for idx, score in combined[:3]:
    print(f"{score:.4f}  {section_chunks[idx].splitlines()[0]}")

Hybrid search gets it right. Now the other direction, to make sure nothing regressed: Chapter 2’s original synonym question, the one embeddings needed to solve in the first place.

holidays_question = "do unused holidays roll over to next year?"
combined_holidays = rrf_combine(holidays_question, section_chunks, bm25)
for idx, score in combined_holidays[:3]:
    print(f"{score:.4f}  {section_chunks[idx].splitlines()[0]}")

Still correct. Hybrid search didn’t have to choose between Chapter 2’s win and this chapter’s win, it keeps both, because each method still contributes its own ranking to the fusion regardless of what the other one thinks.

bare_query = "4477"
print("Embedding scores (all tied at 0.0, meaningless):")
for c in section_chunks:
    print(f"  {cosine_similarity(embed_text(bare_query), embed_text(c)):.3f}  {c.splitlines()[0]}")

print()
combined_bare = rrf_combine(bare_query, section_chunks, bm25)
print("RRF winner:", section_chunks[combined_bare[0][0]].splitlines()[0], "(likely wrong)")

my_question = "who handles password reset issues?"
my_k = 60

combined = rrf_combine(my_question, section_chunks, bm25, k=my_k)
for idx, score in combined[:3]:
    print(f"{score:.4f}  {section_chunks[idx].splitlines()[0]}")

Quick check¶

Before moving on, answer for yourself (no peeking ahead):

Why does the embedding tokenizer losing digits matter so much more for the “4477” question than it would for a typical question about vacation or sick leave?

Why can’t cosine similarity scores and BM25 scores just be added together directly? What does RRF compare instead, and why does that sidestep the problem?

The bare “4477” query broke RRF even though BM25 alone got it right. What specifically went wrong, and is that BM25’s fault, the embedding model’s fault, or RRF’s fault?

If you were building a real system and knew your users mostly asked natural-language questions with the occasional order number or product code mixed in, would you lean more on tuning k, improving the embedding model’s vocabulary, or something else entirely?

Chapter 2 showed embeddings winning where word-matching failed (“holidays” vs “vacation”). This chapter shows the opposite: embeddings losing where keyword matching wins. Does this contradict Chapter 2, or is something else going on?

You can now: combine embeddings and keyword search with Reciprocal Rank Fusion, and you’ve seen a real case where combining them still breaks: a query with no usable signal from one side poisoning the fused ranking.

But one problem remains: every retrieval method in this series so far has worked the same way: embed the question, embed each candidate independently, compare the two fixed vectors after the fact. That misses signal a closer look could catch.

Next → Chapter 8: Re-ranking