How many relevant papers did v1 miss on Tuesday?
Chapter 5 couldn’t answer that, because a missed paper doesn’t appear anywhere in the output. But I have both halves of the answer now: the right answers from Chapter 4’s labels, and every verdict v1 made, because Chapter 5 logged all of them, skips included. This chapter puts them side by side, turns the comparison into two numbers, and uses those numbers for a real decision: does v1 ship?
Where we left off: v1 runs inside its scope and logs every verdict, but “the suggestions look fine” is an impression, and it can’t see missed papers.
What we fix now: Compare every verdict against the labels, measure v1 with precision and recall, set a bar it must clear, and ship it at the “suggest” setting.
By the end, you can:
explain precision and recall in plain words, and why accuracy alone is misleading here
see how a confidence threshold trades one against the other
decide whether v1 is ready to ship, and say what “shipping” means at this autonomy level
The fix: check every verdict against its label¶
An eval (evaluation) runs the system on examples where you know the right answer and counts how often it matches. Most of the work is deciding what to count.
papers = [
{"id": "P01",
"day": "Mon",
"title": "RAG Without Chunking",
"label": True},
{"id": "P02",
"day": "Mon",
"title": "Sparse Mixture-of-Experts at Scale",
"label": False},
{"id": "P03",
"day": "Mon",
"title": "Retrieval Practice Boosts Exam Scores in Engineering Courses",
"label": False},
{"id": "P04",
"day": "Mon",
"title": "Grounding Answers in External Documents",
"label": True},
{"id": "P05",
"day": "Mon",
"title": "Protein Folding with Diffusion Models",
"label": False},
{"id": "P06",
"day": "Mon",
"title": "A Smaller Vision Transformer",
"label": False},
{"id": "P07",
"day": "Mon",
"title": "Citation Accuracy in LLM Answers",
"label": True},
{"id": "P08",
"day": "Mon",
"title": "Faster Training for Speech Models",
"label": False},
{"id": "P09",
"day": "Tue",
"title": "Better Answers with Less Compute",
"label": True},
{"id": "P10",
"day": "Tue",
"title": "Neural Architecture Search for Tiny Devices",
"label": False},
{"id": "P11",
"day": "Tue",
"title": "Dense Retrieval for Product Search",
"label": False},
{"id": "P12",
"day": "Tue",
"title": "Long-Context Models vs. Retrieval: When Is RAG Still Needed?",
"label": True},
{"id": "P13",
"day": "Tue",
"title": "Teaching Robots to Fold Laundry",
"label": False},
{"id": "P14",
"day": "Tue",
"title": "Weather Forecasting with Graph Networks",
"label": False},
{"id": "P15",
"day": "Tue",
"title": "Hallucination Detection via Self-Consistency",
"label": False},
{"id": "P16",
"day": "Tue",
"title": "Reranking Retrieved Passages with Small Models",
"label": True},
{"id": "P17",
"day": "Wed",
"title": "Web-Browsing Agents that Answer from Live Pages",
"label": True},
{"id": "P18",
"day": "Wed",
"title": "Quantizing LLMs to 3 Bits",
"label": False},
{"id": "P19",
"day": "Wed",
"title": "Evaluating RAG Pipelines Without Human Labels",
"label": True},
{"id": "P20",
"day": "Wed",
"title": "Graph Neural Networks for Traffic",
"label": False},
{"id": "P21",
"day": "Wed",
"title": "Chunk Size Matters: A Study of Document Splitting",
"label": True},
{"id": "P22",
"day": "Wed",
"title": "Music Generation with Transformers",
"label": False},
{"id": "P23",
"day": "Wed",
"title": "Code Completion with Repository Context",
"label": True},
{"id": "P24",
"day": "Wed",
"title": "Speeding Up Diffusion Sampling",
"label": False},
]
v1_verdicts = {
"P01": {"confidence": 0.91, "reason": "The paper is about RAG systems and how they index documents."},
"P02": {"confidence": 0.05, "reason": "The paper is about model architecture and training cost, not documents."},
"P03": {"confidence": 0.62, "reason": "The paper is about retrieval, which matches the interest."},
"P04": {"confidence": 0.88, "reason": "The model answers only from fetched passages, which is grounding in documents."},
"P05": {"confidence": 0.03, "reason": "The paper is about proteins, unrelated to documents or answering."},
"P06": {"confidence": 0.04, "reason": "The paper is about image models, not language models answering from documents."},
"P07": {"confidence": 0.81, "reason": "The paper checks whether answers are supported by the passages they cite."},
"P08": {"confidence": 0.06, "reason": "The paper is about speech model training, not documents."},
"P09": {"confidence": 0.4, "reason": "The abstract does not mention documents or retrieval, so it is probably not a match."},
"P10": {"confidence": 0.08, "reason": "The search here is over network designs, not documents."},
"P11": {"confidence": 0.58, "reason": "The paper improves dense retrieval, a core part of RAG."},
"P12": {"confidence": 0.86, "reason": "The paper compares RAG with long-context models for answering from documents."},
"P13": {"confidence": 0.02, "reason": "The paper is about robotics."},
"P14": {"confidence": 0.03, "reason": "The paper is about weather forecasting."},
"P15": {"confidence": 0.35, "reason": "The paper is about hallucinations, but it does not use documents."},
"P16": {"confidence": 0.79, "reason": "Reranking passages before generation is a part of a RAG system."},
"P17": {"confidence": 0.45, "reason": "The abstract is about a browsing agent, which is not clearly about documents."},
"P18": {"confidence": 0.07, "reason": "The paper is about compressing models, not documents."},
"P19": {"confidence": 0.84, "reason": "The paper evaluates RAG pipelines."},
"P20": {"confidence": 0.04, "reason": "The paper is about traffic prediction."},
"P21": {"confidence": 0.77, "reason": "Chunking documents is a part of RAG systems."},
"P22": {"confidence": 0.03, "reason": "The paper is about music generation."},
"P23": {"confidence": 0.66, "reason": "The model writes code using retrieved files from the repository, which is answering from given documents."},
"P24": {"confidence": 0.05, "reason": "The paper is about image generation speed."},
}
by_id = {p["id"]: p for p in papers}
print(f"{len(papers)} labeled papers, {len(v1_verdicts)} v1 verdicts")Every paper ends up in exactly one of four boxes, depending on the label and on what v1 did:
| v1 suggested it | v1 skipped it | |
|---|---|---|
| Relevant | correct suggestion | missed |
| Not relevant | wrong suggestion | correct skip |
The two bold boxes are the mistakes. Chapter 5 could only ever show you wrong suggestions, because they’re on the list. Missed papers were invisible. Here are the boxes filled in:
def outcomes(threshold=0.5):
boxes = {"correct suggestion": [], "wrong suggestion": [], "missed": [], "correct skip": []}
for p in papers:
suggested = v1_verdicts[p["id"]]["confidence"] >= threshold
if p["label"] and suggested:
boxes["correct suggestion"].append(p["id"])
elif not p["label"] and suggested:
boxes["wrong suggestion"].append(p["id"])
elif p["label"] and not suggested:
boxes["missed"].append(p["id"])
else:
boxes["correct skip"].append(p["id"])
return boxes
boxes = outcomes()
for box, ids in boxes.items():
print(f"{box:>18}: {len(ids):>2} {' '.join(ids)}")
print("\nThe mistakes:")
for pid in boxes["wrong suggestion"] + boxes["missed"]:
v = v1_verdicts[pid]
print(f" {pid} ({v['confidence']:.2f}) {by_id[pid]['title']}")
print(f" v1 said: {v['reason']}")That answers Chapter 5’s question: v1 missed P09 on Tuesday and P17 on Wednesday, the two papers whose method only shows up in the full text. The two wrong suggestions are P03 (retrieval practice) and P11 (product search), both “retrieval” in a different sense. Four mistakes out of 24, and none of them look random.
From boxes to numbers¶
Two numbers capture what you care about for a suggestion scout:
Precision: of the papers v1 suggested, how many were actually relevant? This is about trust. Low precision means you learn to ignore its suggestions.
Recall: of the relevant papers, how many did v1 suggest? This is about coverage. Low recall means good papers slip past you, and you won’t even know.
You might expect accuracy, the share of all verdicts that are correct, to be the obvious number. Watch what happens to it here:
def metrics(boxes):
tp, fp = len(boxes["correct suggestion"]), len(boxes["wrong suggestion"])
fn, tn = len(boxes["missed"]), len(boxes["correct skip"])
return {
"precision": tp / (tp + fp) if tp + fp else 0.0,
"recall": tp / (tp + fn) if tp + fn else 0.0,
"accuracy": (tp + tn) / (tp + fp + fn + tn),
}
def show(name, m):
print(f"{name:>18}: precision {m['precision']:.2f} recall {m['recall']:.2f} accuracy {m['accuracy']:.2f}")
show("v1", metrics(outcomes()))
show("skip everything", metrics(outcomes(threshold=2.0))) # confidence never reaches 2
show("suggest everything", metrics(outcomes(threshold=0.0)))A scout that does nothing at all scores 0.58 accuracy, because most papers on any day are irrelevant, so “skip” is usually right. Skipping everything has zero recall, though, and suggesting everything has perfect recall with a precision so low I’d stop reading the list by Wednesday.
v1 lands at 0.80 precision and 0.80 recall: eight of its ten suggestions were right, and it found eight of the ten relevant papers. I don’t report accuracy for the scout at all after this chapter.
The threshold knob¶
v1 suggests a paper when its confidence is at least 0.5. I picked 0.5 because it’s the obvious midpoint, before looking at any results. Lower it and v1 catches more relevant papers but also more wrong ones. Raise it and the list gets cleaner, but more good papers slip through.
threshold = 0.5 # try 0.4, 0.6, 0.7
boxes = outcomes(threshold)
show(f"threshold {threshold}", metrics(boxes))
print(" wrong suggestions:", boxes["wrong suggestion"] or "none")
print(" missed: ", boxes["missed"] or "none")
print("\nFull sweep:")
for t in [0.3, 0.4, 0.5, 0.6, 0.7, 0.8]:
show(f"threshold {t}", metrics(outcomes(t)))Precision and recall pull against each other. At 0.7, every suggestion is right, but v1 finds only seven of the ten relevant papers. At 0.3 it finds all ten, with three wrong suggestions.
0.4 looks like a free lunch: better recall and precision than 0.5. I don’t trust it. It only wins because two relevant papers, P09 and P17, happen to sit at 0.40 and 0.45. Choosing a threshold by staring at the same papers you then measure on is grading your own homework. So 0.5 stays.
Deciding to ship¶
I set the bar before looking at the score, otherwise it’s too easy to set it just below whatever v1 got. For v1 at the “suggest” setting:
Recall at least 0.80. Missing good papers is the costly mistake here: I’d never know they existed.
Precision at least 0.75. A wrong suggestion costs a few seconds, so this bar can sit a little lower, but not so low that I stop trusting the list.
At “act” I’d flip them: a wrong published summary is the expensive mistake, so precision would need to be much higher.
bar_v1 = {"recall": 0.80, "precision": 0.75}
def ready_to_ship(m, bar):
ok = True
for name, minimum in bar.items():
passed = m[name] >= minimum
ok = ok and passed
print(f" {name:>9}: {m[name]:.2f} (needs {minimum:.2f}) -> {'pass' if passed else 'FAIL'}")
return ok
m = metrics(outcomes(0.5))
print("Ship v1 at 'suggest'?", "YES" if ready_to_ship(m, bar_v1) else "NO")v1 clears both bars, just, so it ships. At this autonomy level, shipping is modest: v1 runs every morning on new papers, posts its suggestions to me, and writes every verdict and trace to its log. It still can’t publish or change anything.
The log matters more than the shipping. From the next chapter on, the course works with what v1 does in real use instead of our hand-made week.
next_week = [
{"id": "N01", "title": "A Survey of Retrieval-Augmented Generation",
"confidence": 0.93, "reason": "The paper surveys RAG systems."},
{"id": "N02", "title": "Tuning the Retrieval Stage of Video Recommenders",
"confidence": 0.71, "reason": "The paper improves a retrieval stage, which matches the interest."},
{"id": "N03", "title": "Answering Legal Questions from Court Rulings",
"confidence": 0.48, "reason": "The paper is about law, which seems outside the interest."},
{"id": "N04", "title": "Sparse Attention for Long Videos",
"confidence": 0.06, "reason": "The paper is about video models."},
]
for p in next_week:
verdict = "suggest" if p["confidence"] >= 0.5 else "skip"
print(f"{p['id']} {verdict:>7} ({p['confidence']:.2f}) {p['title']}")
print(f" {p['reason']}")There are no labels here, so there’s no precision or recall to compute. In real use there never are, unless someone makes them. The eval number describes the 24 papers I labeled, not the papers v1 sees tomorrow.
N02 and N03 are the more worrying part. A recommender’s “retrieval stage” is the P11 trap again, in a domain the dataset never covered. N03 answers legal questions from court rulings, which is clearly relevant, but v1’s reason suggests it judged the topic (law) instead of the method (answering from documents). Neither kind of paper is in my 24, so the eval couldn’t have warned me about either.
An eval only contains what its author thought of. To see how v1 does on the papers it actually meets, I have to measure it on real use, which is where Continuous Calibration starts.
Quick check¶
Before moving on, answer for yourself (no peeking ahead):
In your own words, what’s the difference between precision and recall for the scout? Which mistake does each one count?
Check your reasoning ▸
Precision asks: of what the scout suggested, how much was relevant? It counts wrong suggestions. Recall asks: of what was relevant, how much did the scout suggest? It counts missed papers. A strong answer ties them to the reader’s experience: precision is whether you can trust the list, recall is whether good papers slip past you unseen.
A scout that skips every paper gets 0.58 accuracy. Why is accuracy a poor number for this task?
Check your reasoning ▸
Because most papers are irrelevant, so “skip” is usually correct, and accuracy rewards a scout that does nothing. A good answer notes that accuracy hides the two mistakes that actually matter here, missed papers and wrong suggestions, inside one blended number.
Threshold 0.4 beat 0.5 on both precision and recall. Why didn’t we switch to it?
Check your reasoning ▸
Because 0.4 was chosen by looking at the same 24 papers it was measured on. It’s better mostly because two specific papers sit at 0.40 and 0.45. Tuning to your eval set makes the score look better than the scout will be on new papers. A strong answer adds that the fix is more (or separate) labeled data, not a cleverer threshold.
Why does the recall bar matter more than the precision bar at the “suggest” setting, and how would that change at “act”?
Check your reasoning ▸
At “suggest”, a wrong suggestion costs you seconds, but a missed paper is gone for good, so recall is the costlier failure. At “act”, a wrong decision becomes a public post, so precision becomes the costly failure and its bar would need to be much higher. The right bar depends on the autonomy level.
v1 passed its bar with 0.80 and 0.80. Why doesn’t that mean it will be right 80% of the time next week?
Check your reasoning ▸
The score describes the 24 labeled papers, which only contain the kinds of papers you thought to include. Next week brings papers unlike any of them, like recommender retrieval or legal question answering, and there are no labels to check against. A strong answer says the eval is a starting estimate, and that measuring on real use is the only way to see how v1 actually performs.
You can now: evaluate a scout against labeled data with precision and recall, see how a threshold trades one for the other, set a bar that fits the autonomy level, and ship v1 at “suggest”.
But one problem remains: the eval measured v1 on papers you chose and labeled. Real use brings papers you never thought of, with no labels at all. The next chapter starts Continuous Calibration: scoring v1 on a week of real runs from its log, and seeing how the numbers hold up.
Next → Chapter 7: Run Evals on Real Use