Every retrieval method since Chapter 2 has worked the same way: embed the question, embed each candidate independently, compare the two fixed vectors. That’s a bi-encoder, “bi” because the question and the candidate never interact, they’re each squeezed into a vector on their own, then compared afterward. A cross-encoder does something more expensive: it looks at the question and a specific candidate together, as one input, and can notice things a bi-encoder’s after-the-fact comparison misses.
This chapter builds a small, honest version of that idea from scratch, and the real architectural pattern that makes it practical: re-rank a short list instead of scoring everything the expensive way.
Where we left off: Every retrieval method since Chapter 2 has worked the same way: embed the question, embed each candidate separately, compare the two fixed vectors afterward.
What we fix now: Build a small, honest approximation of interaction-based re-ranking, and the real architectural pattern that makes it practical: re-rank a short list instead of scoring everything the expensive way.
By the end, you can:
explain the difference between a bi-encoder and a cross-encoder, and why one is precomputable and the other isn’t
build a late-interaction score from scratch and see it fix a dilution failure a bi-encoder can’t
combine cheap and expensive retrieval into a two-stage pipeline
Source
import numpy as np
import re
import time
# Same setup as Chapters 2-7 (including Chapter 7's IT Support section).
training_corpus = [
"Employees may carry over unused vacation days into the next calendar year.",
"Staff can roll over unused holiday days into next year.",
"Workers get paid time off and vacation days every year.",
"Most employees take a vacation or holiday once a year.",
"Unused vacation time can roll over to next year for many employees.",
"Holiday leave and vacation leave are treated the same way at most companies.",
"Some employees prefer to use their holiday days instead of saving them.",
"Vacation days and holiday days both count as paid time off.",
"Expense reports must be submitted within days of the purchase.",
"Employees submit an expense report after every business purchase.",
"The finance team reviews each expense report and each purchase receipt.",
"A purchase over the limit needs approval before the expense is reimbursed.",
"Expense claims and purchase receipts are reviewed together by finance.",
"Remote employees receive a home office stipend to set up their workspace.",
"The company gives remote workers a stipend for home office equipment.",
"Employees working remote from home can request an office stipend.",
"A stipend helps remote staff pay for office furniture at home.",
"The office stipend is a one time payment for remote employees.",
"Vacation policy, expense policy, and remote work policy are all in the handbook.",
"Every employee should read the handbook for vacation and expense rules.",
"Employees who are sick should notify their manager as soon as possible.",
"Sick leave is paid time off for employees who are too unwell to work.",
"A doctor's note may be required after three consecutive sick days.",
"Unused sick leave does not roll over into the next year.",
"Employees should stay home and rest when they are sick.",
"Standard working hours are nine to five, Monday through Friday.",
"Employees may request a flexible schedule with manager approval.",
"Overtime must be approved in advance by a manager.",
"Most staff work standard business hours unless a flexible schedule is approved.",
"Employees are expected to treat colleagues and customers with respect.",
"Harassment or discrimination of any kind will not be tolerated.",
"Violations of the code of conduct may result in disciplinary action.",
"Every employee must follow the code of conduct and treat others with respect.",
"Company laptops must be encrypted and password protected.",
"Employees should not install unauthorized software on company devices.",
"Report a lost or stolen laptop to IT immediately.",
"IT equipment must be returned when an employee leaves the company.",
"The IT team manages laptops, software, and device security for the company.",
"Contact the helpdesk if you have a technical problem with your computer.",
"The support team can help reset passwords and fix login issues.",
"Call the helpdesk for any urgent technical problem during business hours.",
]
def tokenize(text):
return re.findall(r"[a-z']+", text.lower())
tokens_per_sentence = [tokenize(s) for s in training_corpus]
vocab = sorted(set(w for sent in tokens_per_sentence for w in sent))
word_index = {w: i for i, w in enumerate(vocab)}
V = len(vocab)
window = 4
cooc = np.zeros((V, V))
for sent in tokens_per_sentence:
for i, w in enumerate(sent):
for j in range(max(0, i - window), min(len(sent), i + window + 1)):
if i != j:
cooc[word_index[w], word_index[sent[j]]] += 1
X = np.log1p(cooc)
U, S, Vt = np.linalg.svd(X, full_matrices=False)
DIM = 25
word_vectors = U[:, :DIM] * S[:DIM]
def embedding(word):
return word_vectors[word_index[word]] if word in word_index else None
def embed_text(text):
words = [w for w in tokenize(text) if w in word_index]
if not words:
return np.zeros(DIM)
return np.mean([embedding(w) for w in words], axis=0)
def cosine_similarity(a, b):
norm = np.linalg.norm(a) * np.linalg.norm(b)
return float(np.dot(a, b) / norm) if norm else 0.0
handbook_document = """Vacation Policy
Employees may carry over up to 5 unused vacation days into the next calendar year. Time off should be requested from a manager at least two weeks in advance when possible.
Sick Leave Policy
Employees who are too unwell to work should notify their manager as soon as possible. Sick leave is paid time off and does not roll over into the next year. A doctor\'s note may be required after three consecutive sick days.
Expense Policy
Expense reports must be submitted within 30 days of the purchase date. The finance team reviews each expense report and matching purchase receipt before reimbursement.
Remote Work Policy
Remote employees are provided a one-time home office stipend of 500 euros. The stipend covers office furniture and basic equipment needed to work from home.
Working Hours
Standard working hours are nine to five, Monday through Friday. Employees may request a flexible schedule with manager approval, and any overtime must be approved in advance.
Code of Conduct
Employees are expected to treat colleagues and customers with respect. Harassment or discrimination of any kind will not be tolerated, and violations may result in disciplinary action.
IT Support
For urgent technical problems, contact the helpdesk at extension 4477. For non-urgent requests, submit a ticket through the internal support portal."""
section_chunks = [p.strip() for p in handbook_document.split("\n\n")]
def bi_encoder_score(question, candidate):
return cosine_similarity(embed_text(question), embed_text(candidate))
print(f"Ready: {len(section_chunks)} sections, {V} words in vocabulary.")# The exact mismatch from Chapter 5, revisited.
question = "how many sick days can be carried over to next year?"
ranked = sorted(section_chunks, key=lambda c: bi_encoder_score(question, c), reverse=True)
for c in ranked[:3]:
print(f"{bi_encoder_score(question, c):.3f} {c.splitlines()[0]}")Same wrong answer as Chapter 5: Vacation Policy first, Sick Leave Policy second. But notice the correct answer isn’t missing, it’s sitting right there at rank 2. Averaging every word in the question and every word in each candidate into one vector each washes out exactly the one word that mattered most: “sick.” It’s still in there somewhere, contributing to the average, just outweighed by everything else.
Looking at words, not just averages¶
Worth being precise before building anything: what follows is not a cross-encoder. A real cross-encoder performs joint neural encoding of the question and a candidate together, one forward pass through a trained network, and that’s what the Colab bonus at the end of this chapter actually runs. The lightweight method built below demonstrates the broader motivation for interaction-based re-ranking using a late interaction style score instead (the idea behind models like ColBERT), built from nothing more than the word vectors already on hand: keep each word’s vector separate instead of collapsing the whole question into one, and for every question word, find its single best-matching word anywhere in the candidate, then average those best-matches.
def late_interaction_score(question, candidate):
q_words = [w for w in tokenize(question) if w in word_index]
c_words = [w for w in tokenize(candidate) if w in word_index]
if not q_words or not c_words:
return 0.0
candidate_vectors = [embedding(w) for w in c_words]
total = 0.0
for qw in q_words:
qv = embedding(qw)
best_match = max(cosine_similarity(qv, cv) for cv in candidate_vectors)
total += best_match
return total / len(q_words)
ranked = sorted(section_chunks, key=lambda c: late_interaction_score(question, c), reverse=True)
for c in ranked[:3]:
print(f"{late_interaction_score(question, c):.3f} {c.splitlines()[0]}")Sick Leave Policy wins now. The word “sick” in the question finds its best match against the word “sick” inside the Sick Leave Policy section, a near-perfect single-word match, and that strong signal survives instead of getting averaged away with everything else. Chapters 5 and 7 already fixed this same mismatch two different ways (citations that reveal it, BM25’s exact keyword match), this is a third, genuinely different mechanism: not exact words, not manual metadata, just refusing to blur word-level detail together too early.
Same question: “how many sick days can be carried over to next year?”
Top result: Vacation Policy at 0.930, the wrong section. The word “sick” is in there somewhere, just outweighed once averaged with everything else in the question.
Same question: “how many sick days can be carried over to next year?”
Top result: Sick Leave Policy at 0.933, correct. “Sick” finds its own near-perfect match inside the Sick Leave section instead of being blurred away.
Why this isn’t the new default¶
If this works better, why not always compare word-by-word instead of averaging? Because it’s a lot more expensive. Averaging lets you precompute one vector per candidate once, ever, and every future query just compares against that cached vector. Word-by-word comparison has to recompute something for every single question against every single candidate, every time.
start = time.perf_counter()
for _ in range(200):
for c in section_chunks:
bi_encoder_score(question, c)
bi_ms = (time.perf_counter() - start) / 200 * 1000
start = time.perf_counter()
for _ in range(200):
for c in section_chunks:
late_interaction_score(question, c)
li_ms = (time.perf_counter() - start) / 200 * 1000
print(f"bi-encoder (precomputable): {bi_ms:.3f} ms for all {len(section_chunks)} sections")
print(f"late interaction (per-query): {li_ms:.3f} ms for all {len(section_chunks)} sections")
print(f"ratio: {li_ms / bi_ms:.1f}x slower")Already slower by a wide margin, at just seven candidates. Scale that up to thousands or millions of chunks (Chapter 6’s problem, back again) and running the expensive comparison against everything stops being realistic. The practical answer: use the cheap method to narrow the field first, and only spend the expensive comparison on a short list.
K = 3
# Stage 1: fast bi-encoder search narrows the field.
stage_1 = sorted(section_chunks, key=lambda c: bi_encoder_score(question, c), reverse=True)[:K]
print(f"Stage 1 shortlist (top {K}, bi-encoder):")
for c in stage_1:
print(f" {bi_encoder_score(question, c):.3f} {c.splitlines()[0]}")
# Stage 2: expensive re-ranking only touches the shortlist.
stage_2 = sorted(stage_1, key=lambda c: late_interaction_score(question, c), reverse=True)
print(f"\nStage 2 (re-ranked):")
for c in stage_2:
print(f" {late_interaction_score(question, c):.3f} {c.splitlines()[0]}")Two-stage retrieval in miniature: cheap and broad, then expensive and narrow. The final answer is correct, and the expensive step only ever ran three times instead of seven, a small saving here, a decisive one once “seven” becomes “seven million.”
extension_question = "what department is 4477"
ranked = sorted(section_chunks, key=lambda c: late_interaction_score(extension_question, c), reverse=True)
for c in ranked[:3]:
print(f"{late_interaction_score(extension_question, c):.3f} {c.splitlines()[0]}")
print("\n(Still wrong: late interaction is still built on the same limited")
print(" embedding vocabulary. This is exactly why Chapter 7's hybrid search")
print(" and this chapter's re-ranking solve different problems, not")
print(" competing solutions to the same one.)")K = 2
stage_1 = sorted(section_chunks, key=lambda c: bi_encoder_score(question, c), reverse=True)[:K]
stage_2 = sorted(stage_1, key=lambda c: late_interaction_score(question, c), reverse=True)
for c in stage_2:
print(f"{late_interaction_score(question, c):.3f} {c.splitlines()[0]}")Bonus (Colab only): a real cross-encoder¶
The hand-built late_interaction_score above is a real, honest
technique, but a real cross-encoder is a neural network trained
specifically to score a (question, candidate) pair, and it’s
noticeably better in practice. This cell runs an actual one on the
same shortlist:
# Colab only: downloads a real cross-encoder model.
!pip install -q -U sentence-transformers
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
shortlist = sorted(section_chunks, key=lambda c: bi_encoder_score(question, c), reverse=True)[:3]
pairs = [(question, c) for c in shortlist]
scores = reranker.predict(pairs)
for score, c in sorted(zip(scores, shortlist), reverse=True):
print(f"{float(score):.3f} {c.splitlines()[0]}")Quick check¶
Before moving on, answer for yourself (no peeking ahead):
In your own words, what’s the actual difference between a bi-encoder and a cross-encoder, and why does that difference make one precomputable and the other not?
Check your reasoning ▸
A bi-encoder embeds the question and each candidate completely separately, so every candidate’s vector can be computed once, in advance, and just compared at query time. A cross-encoder looks at the question and a specific candidate together as one joint input, so the comparison itself depends on the question, meaning nothing about it can be precomputed ahead of time.
Re-ranking only helps if the right answer is already somewhere in the shortlist. What would you expect to happen if
K=1and the bi-encoder’s single top pick is wrong?
Check your reasoning ▸
If the bi-encoder’s single top pick is wrong, re-ranking can’t help at all, there’s nothing else in the shortlist to promote instead. This shows re-ranking only fixes ordering within a shortlist that already contains the right answer; it can’t recover an answer that never made the list in the first place.
This chapter’s example, Chapter 7’s example, and Chapter 5’s example all involve retrieval getting confused between Vacation Policy and Sick Leave Policy or similar near-neighbors. Why do you think that specific confusion keeps coming back across so many different fixes, rather than being solved once and for all?
Check your reasoning ▸
A strong answer recognizes each fix addresses a different mechanism, not the same root cause: citations make the mistake visible without preventing it, hybrid search helps only when there’s a distinctive exact token to latch onto, and re-ranking helps only when the shortlist already contains the right answer. None of them changes the underlying embedding model’s tendency to blend similar-vocabulary sections together, so the confusion resurfaces wherever that root cause is still in play.
If you had to combine Chapter 7’s hybrid search and this chapter’s re-ranking into one pipeline, where would each one go, and in what order? Why?
Check your reasoning ▸
The natural order is hybrid search first, to build a broad, high-recall shortlist cheaply by combining keyword and embedding signal, then re-ranking second, to carefully reorder just that short list with the more expensive word-level comparison. Running it the other way, expensive re-ranking over the full candidate set before narrowing, would throw away exactly the speed benefit the two-stage pattern is designed to provide.
Chapter 7’s “4477” question failed because the word never existed in the embedding vocabulary at all. Would this chapter’s late-interaction re-ranking fix that specific failure? Why or why not?
Check your reasoning ▸
No, and the chapter says so directly: late interaction still relies on word vectors that exist in the vocabulary, comparing word-by-word instead of averaging doesn’t help if one side has no vector to compare in the first place. This is a good example of a fix with a specific, bounded scope: re-ranking fixes dilution, not vocabulary gaps, and confusing the two would mean expecting a fix to work somewhere it fundamentally can’t.
You can now: explain the difference between a bi-encoder and a cross-encoder, and build a two-stage pipeline that cheaply narrows the field before spending an expensive comparison on just the shortlist.
But one problem remains: eight chapters, four retrieval techniques, and every single case for “this one is better” has been one or two hand-picked examples. That’s anecdote, not measurement.
Next → Chapter 9: Evaluation