v1 has been running for a week. Every morning it read the new papers, posted its suggestions, and wrote every verdict to its log. Nothing exploded, and it never published anything, because it can’t. I opened a few of its suggestions and ignored a couple that looked off.
That’s an impression, not a measurement. Chapter 6 said 0.80 and 0.80, on 24 papers I picked myself. This chapter checks what those numbers look like on the papers v1 actually met. It’s the start of Continuous Calibration, the second half of the course.
Where we left off: v1 passed its eval and shipped at “suggest”, but the eval only covers papers we chose and labeled ourselves.
What we fix now: Measure v1 on its first real week: turn its log into something you can score, and compare the real numbers with the eval’s.
By the end, you can:
name three ways to get labels for real use, and what each one can’t see
score a real week from the log, including a problem the eval never measured
explain why a drop in the numbers doesn’t tell you what to fix
The fix: labels for real use¶
The eval worked because every paper had a label. Real use doesn’t come with labels, so they have to come from somewhere:
Spot-check. Take papers from the log and label them yourself, with Chapter 4’s guideline. Slow, but the most trustworthy option, and the only one that sees both kinds of mistake: wrong suggestions and missed papers.
Implicit signals. Record what you actually do: which suggestions you open, which you ignore. Free, but it only sees what v1 showed you. A missed paper never gets clicked.
A judge model. A second model labels papers with the guideline. It scales, but then you need an eval for the judge. I left this out of the course on purpose; it doubles the number of things that can be wrong, and the scout is small enough that I can label by hand.
I’ll use a spot-check. v1’s first week has 20 entries, few enough to label all of them. With a busier feed I’d label a random sample, and there’s a cell below showing what that does to the numbers.
week = [
{"id": "W01",
"day": "Mon",
"title": "RAG for Clinical Guidelines",
"abstract": "We let a language model answer doctors' questions from retrieved clinical guidelines. Accuracy rises by 12 percent over the model alone.",
"full_text": "Guideline sections are retrieved by similarity and the model must answer only from them, citing the section it used.",
"conf_v1": 0.88,
"full_v1": False,
"reason_v1": "The model answers from retrieved clinical guidelines, and accuracy rises by 12 percent."},
{"id": "W02",
"day": "Mon",
"title": "Tuning the Retrieval Stage of Video Recommenders",
"abstract": "We tune the retrieval stage that picks candidate videos for a recommender. Watch time rises by 3 percent.",
"full_text": "The retrieval stage selects 500 candidate videos, which a ranking model then orders. No text is generated.",
"conf_v1": 0.71,
"full_v1": False,
"reason_v1": "The paper improves a retrieval stage, which matches the interest."},
{"id": "W03",
"day": "Mon",
"title": "Answering Legal Questions from Court Rulings",
"abstract": "We answer tax law questions with a system that consults court rulings. Lawyers rated 78 percent of the answers as correct.",
"full_text": "Relevant rulings are retrieved for each question, and the language model writes its answer only from those rulings, quoting them.",
"conf_v1": 0.48,
"full_v1": False,
"reason_v1": "The paper is about law, which seems outside the interest."},
{"id": "W04",
"day": "Mon",
"title": "Sparse Attention for Long Videos",
"abstract": "We make video transformers attend to only 10 percent of frame patches. Speed doubles with little accuracy loss.",
"full_text": "A learned mask selects which patches each frame attends to.",
"conf_v1": 0.06,
"full_v1": False,
"reason_v1": "The paper is about video models."},
{"id": "W05",
"day": "Tue",
"title": "A Survey of Retrieval-Augmented Generation",
"abstract": "We review more than 200 papers on retrieval-augmented generation and group them by retriever, generator, and training method.",
"full_text": "The survey covers 212 papers published over four years and ends with open problems in evaluation.",
"conf_v1": 0.93,
"full_v1": False,
"reason_v1": "The paper surveys over 300 RAG papers."},
{"id": "W06",
"day": "Tue",
"title": "Memory Retrieval During Sleep",
"abstract": "We study how the brain retrieves memories during sleep in 40 volunteers. Retrieval events predicted next-day recall.",
"full_text": "Volunteers' brain activity was recorded overnight, and a language model was used only to transcribe their morning dream reports.",
"conf_v1": 0.64,
"full_v1": False,
"reason_v1": "The paper studies retrieval, which matches the interest."},
{"id": "W07",
"day": "Tue",
"title": "Faster Tokenizers",
"abstract": "We speed up text tokenization by 4 times with a new merge table layout.",
"full_text": "Merge rules are stored in a flat array for cache-friendly lookups.",
"conf_v1": 0.05,
"full_v1": False,
"reason_v1": "The paper is about tokenizer speed."},
{"id": "W08", "day": "Tue", "repeat_of": "W01"},
{"id": "W09",
"day": "Wed",
"title": "Citations That Point to the Right Sentence",
"abstract": "We train a model to cite the exact sentence that supports each claim in its answer. Citation precision rises from 61 to 83 percent.",
"full_text": "Answers are generated from retrieved passages, and each claim is linked to one supporting sentence.",
"conf_v1": 0.83,
"full_v1": False,
"reason_v1": "The paper improves citations in answers generated from passages."},
{"id": "W10",
"day": "Wed",
"title": "Protein Language Models",
"abstract": "We train a language model on 50 million protein sequences. It predicts protein function better than previous models.",
"full_text": "Protein sequences are treated as text and the model learns by predicting masked amino acids.",
"conf_v1": 0.04,
"full_v1": False,
"reason_v1": "The paper is about proteins."},
{"id": "W11",
"day": "Wed",
"title": "Coding Agents That Read the Docs First",
"abstract": "We improve coding agents on 3 benchmarks with a simple change to their workflow.",
"full_text": "Before writing code, the agent fetches the relevant API documentation pages and writes its code from them.",
"conf_v1": 0.42,
"full_v1": False,
"reason_v1": "The paper is about coding agents, not documents."},
{"id": "W12", "day": "Wed", "repeat_of": "W05"},
{"id": "W13",
"day": "Thu",
"title": "Query Rewriting for Better Retrieval in RAG",
"abstract": "We rewrite user questions before retrieval in a RAG system. Recall rises by 14 points and answer accuracy by 6 points.",
"full_text": "A small model rewrites each question into a search query, and the answer is generated from the passages retrieved with it.",
"conf_v1": 0.85,
"full_v1": False,
"reason_v1": "Query rewriting raises recall by 40 points in a RAG system."},
{"id": "W14",
"day": "Thu",
"title": "Retrieval-Induced Forgetting in Language Learners",
"abstract": "Recalling some words made 120 language learners forget related words. The effect lasted one week.",
"full_text": "This is a psychology experiment with human learners and vocabulary cards. No computer models are used.",
"conf_v1": 0.6,
"full_v1": False,
"reason_v1": "The paper is about retrieval and language."},
{"id": "W15",
"day": "Thu",
"title": "Graph Transformers for Molecules",
"abstract": "We apply graph transformers to predict molecule properties. Error falls by 9 percent.",
"full_text": "Atoms are nodes and bonds are edges.",
"conf_v1": 0.03,
"full_v1": False,
"reason_v1": "The paper is about chemistry."},
{"id": "W16",
"day": "Thu",
"title": "Low-Resource Machine Translation",
"abstract": "We translate between 12 low-resource languages with a single model. Quality rises by 3 BLEU points.",
"full_text": "The model is trained on back-translated data.",
"conf_v1": 0.09,
"full_v1": False,
"reason_v1": "The paper is about translation."},
{"id": "W17",
"day": "Fri",
"title": "Faithful Summaries of Meeting Transcripts",
"abstract": "We generate meeting summaries that stay faithful to what was said. Human raters preferred them 70 percent of the time.",
"full_text": "For each topic, the relevant transcript passages are retrieved and the model writes the summary only from them.",
"conf_v1": 0.45,
"full_v1": False,
"reason_v1": "The paper is about meeting summaries, not documents."},
{"id": "W18",
"day": "Fri",
"title": "Hybrid Search for RAG over Product Manuals",
"abstract": "We combine keyword and dense retrieval to answer questions from product manuals. Answer accuracy rises by 9 points.",
"full_text": "Keyword and dense results are merged, and the model answers from the top passages.",
"conf_v1": 0.87,
"full_v1": True,
"reason_v1": "The paper answers questions from manuals using hybrid retrieval."},
{"id": "W19",
"day": "Fri",
"title": "Speculative Decoding at Scale",
"abstract": "We speed up text generation by 2.5 times with a small draft model.",
"full_text": "The draft model proposes tokens that the large model verifies in one pass.",
"conf_v1": 0.05,
"full_v1": False,
"reason_v1": "The paper is about generation speed."},
{"id": "W20", "day": "Fri", "repeat_of": "W09"},
]
# Repeats are the same paper posted again (for example, an updated version).
_originals = {p['id']: p for p in week if 'repeat_of' not in p}
week = [dict(_originals[p['repeat_of']], id=p['id'], day=p['day'], repeat_of=p['repeat_of'])
if 'repeat_of' in p else dict(p, repeat_of=None) for p in week]
for day in ["Mon", "Tue", "Wed", "Thu", "Fri"]:
print(f"--- {day} ---")
for p in [p for p in week if p["day"] == day]:
verdict = "suggest" if p["conf_v1"] >= 0.5 else "skip"
print(f" {p['id']} {verdict:>7} ({p['conf_v1']:.2f}) {p['title']}")Look at the titles across the days. RAG for Clinical Guidelines shows up on Monday and again on Tuesday, the survey on Tuesday and Wednesday, the citations paper on Wednesday and Friday. Papers get re-posted, for example when authors upload a new version, and v1 doesn’t remember what it has already seen. My eval couldn’t have caught this: every paper in it was unique.
Now the spot-check. Below are the labels for the 17 unique papers, made with Chapter 4’s guideline. (Try labeling a few yourself before you read them: W03, W06, W11 and W17 are the interesting ones.)
spot_check = {
"W01": True,
"W02": False,
"W03": True,
"W04": False,
"W05": True,
"W06": False,
"W07": False,
"W09": True,
"W10": False,
"W11": True,
"W13": True,
"W14": False,
"W15": False,
"W16": False,
"W17": True,
"W18": True,
"W19": False
}
def score(entries, labels, threshold=0.5):
boxes = {"correct suggestion": [], "wrong suggestion": [], "missed": [], "correct skip": []}
for p in entries:
if p["repeat_of"] or p["id"] not in labels:
continue
suggested = p["conf_v1"] >= threshold
relevant = labels[p["id"]]
key = ("correct suggestion" if suggested else "missed") if relevant else \
("wrong suggestion" if suggested else "correct skip")
boxes[key].append(p["id"])
tp, fp, fn = len(boxes["correct suggestion"]), len(boxes["wrong suggestion"]), len(boxes["missed"])
precision = tp / (tp + fp) if tp + fp else 0.0
recall = tp / (tp + fn) if tp + fn else 0.0
return boxes, precision, recall
boxes, precision, recall = score(week, spot_check)
for box, ids in boxes.items():
print(f"{box:>18}: {len(ids):>2} {' '.join(ids)}")
repeats = [p["id"] for p in week if p["repeat_of"] and p["conf_v1"] >= 0.5]
print(f"\nRepeated suggestions (same paper suggested again): {len(repeats)} {' '.join(repeats)}")
print(f"\n{'':>14} precision recall")
print(f"{'eval (Ch 6)':>14} 0.80 0.80")
print(f"{'real week':>14} {precision:.2f} {recall:.2f}")
print(f"{'bar to ship':>14} 0.75 0.80")That’s a real drop. On the papers it actually met, v1 got five of its eight suggestions right and found five of the eight relevant papers. Both are below the bar it had to clear to ship. It also suggested three papers I’d already been shown, which neither precision nor recall counts.
The eval wasn’t wrong. It measured 24 papers that I picked, and real use has more kinds of papers and more kinds of situations than I thought to include.
How many labels are enough?¶
We labeled everything. In practice you’d often label a sample, a day or two from a busy week. That saves time, but a small sample gives a jumpy number.
sample_days = ["Mon", "Tue"]
sample = [p for p in week if p["day"] in sample_days]
_, p_sample, r_sample = score(sample, spot_check)
labeled = sum(1 for p in sample if not p["repeat_of"])
print(f"Labeled {labeled} papers from {sample_days}: precision {p_sample:.2f}, recall {r_sample:.2f}")With only a handful of labeled papers, a single paper swings the number a lot. Labeling Friday alone gives precision 1.00 and recall 0.50, and labeling Thursday alone gives the reverse. Neither is “the” number. So: label enough papers, and watch the trend over several weeks instead of reacting to one small sample.
What about the free signal, the suggestions I actually opened? Say I opened the five good suggestions and ignored the three odd ones:
opened = {"W01", "W05", "W09", "W13", "W18"}
suggested = [p["id"] for p in week if not p["repeat_of"] and p["conf_v1"] >= 0.5]
click_precision = len(opened & set(suggested)) / len(suggested)
print(f"Suggested: {len(suggested)}, opened: {len(opened)}")
print(f"Precision from clicks: {click_precision:.2f}")
print("Recall from clicks: impossible, a missed paper is never shown, so it can't be clicked")Clicks give a rough precision for free, which is useful for watching trends between spot-checks. They can’t see missed papers at all. W03, W11 and W17 were never on my screen, so nothing I clicked can reveal them. Only labeling does.
for t in [0.40, 0.45, 0.50, 0.60, 0.65, 0.70]:
_, p_t, r_t = score(week, spot_check, threshold=t)
ok = p_t >= 0.75 and r_t >= 0.80
print(f"threshold {t:.2f}: precision {p_t:.2f} recall {r_t:.2f} {'passes' if ok else ''}")No threshold works. Lower it and v1 catches the missed papers but lets in more wrong ones. Raise it and the list gets cleaner but misses more. The threshold only moves the cut-off; it can’t change what v1 thinks of each paper.
Two numbers told me v1 got worse. They don’t tell me why. Is the prompt unclear? Is a tool missing? Is it deciding too fast? For that I have to open the individual mistakes.
Quick check¶
Before moving on, answer for yourself (no peeking ahead):
Name the three ways to get labels for real use from this chapter, and one thing each of them can’t do well.
Check your reasoning ▸
Spot-checking is slow and doesn’t scale to every paper. Implicit signals like clicks only see what v1 showed you, so they can never reveal missed papers. A judge model scales, but you then need to evaluate the judge itself. A strong answer adds that spot-checks are the most trustworthy baseline to compare the others against.
The eval said 0.80 and 0.80, the real week says 0.62 and 0.62. Was the eval wrong?
Check your reasoning ▸
Not wrong, just narrower than real use. It correctly measured v1 on the 24 papers we chose. Real use contains kinds of papers and situations (like repeats) that weren’t in those 24. A good answer says the eval is a starting estimate, and real-use measurement is what keeps it honest.
Why don’t repeated suggestions show up in precision or recall?
Check your reasoning ▸
Because they’re counted per unique paper, and a repeated suggestion is correct each time: the paper really is relevant. The problem is a different one, wasting your attention on something you’ve already seen. A strong answer says every metric is blind to the problems it wasn’t designed to count, so you need to keep looking at the raw log too.
You labeled only Friday and got a very different number from the full week. What does that tell you about sampling?
Check your reasoning ▸
That small samples are noisy: with few labeled papers, a single paper moves the estimate a lot. The answer isn’t to avoid sampling but to label enough papers and look at trends across weeks rather than trusting one small snapshot.
No threshold got v1 over both bars. Why can’t the threshold fix this, and what would you look at instead?
Check your reasoning ▸
The threshold only decides where to cut v1’s confidences; it can’t change which papers v1 is confident about. If v1 is confidently wrong about some papers and unsure about good ones, no cut separates them. You’d look at the individual mistakes, their traces and reasons, to find what they have in common.
You can now: measure a shipped agent on real use with a spot-check, spot a problem the eval never counted, and explain why a noisy sample or a single threshold can mislead you.
But one problem remains: the numbers say v1 got worse in real use, but not why. The next chapter opens up the individual mistakes, reads their traces, and groups them into patterns you can actually fix.
Next → Chapter 8: Find Error Patterns