Four patterns from Chapter 8, four fixes:
A. Deciding from the abstract alone (missed papers)
B. Falling for the wrong kind of “retrieval” (wrong suggestions)
C. Suggesting the same paper again (wasted attention)
D. Putting numbers in the reason that aren’t in the paper (misleading reasons)
None of the fixes is hard to write. What I’m careful about is the order: one fix at a time, and after each one I re-measure everything, including the 24 labeled papers from Chapter 4 that v1 already handled well. A fix that repairs this week and quietly breaks last week is a trade I’d rather know I’m making.
Where we left off: We know v1’s four error patterns, and that fixes can interfere with each other.
What we fix now: Apply one fix per pattern, re-measure after every step on both the real week and the labeled week, and catch a fix that breaks something.
By the end, you can:
decide when a step should be fixed code (a workflow) instead of the model’s choice
apply fixes one at a time with a scorecard, and spot a regression
explain why a system can look better on every metric you track and still get worse
Four fixes, one per pattern¶
| Pattern | Fix | Where the fix lives |
|---|---|---|
| A. Decided from the abstract alone | Full-text step: after a quick screen of the abstract, any paper that isn’t clearly off-topic gets its full text read, always | code (a fixed workflow step) |
| B. Wrong kind of retrieval | Guideline in the prompt: give the model Chapter 4’s labeling guideline, word for word | prompt |
| C. Suggested again | Memory: remember what was already suggested and skip it | code |
| D. Invented numbers | Grounding check: run Chapter 8’s invented_numbers on every reason; withhold any reason that fails | code (a guardrail) |
Fix A is the one I thought about longest. In v1, the model decided whether to read the full text, and it almost never did, which caused both A and B. So I take that choice away from it. If a step should happen every time, plain code can do it, the same way every time. The model keeps the judgment calls; the code keeps the routine.
That makes the scout less of an agent in the Chapter 1 sense, and I’m fine with that. I’d rather have a fixed workflow with a model at the few points that need judgment than a free agent that skips the step that matters. Roughly how I decide:
| Use plain code when... | Let the model decide when... |
|---|---|
| the step should always happen | the right step depends on what it just saw |
| the rule is known and exact | the input needs interpreting |
| a mistake would be expensive | a mistake is cheap and caught later |
My default is the least autonomous design that solves the problem. I only hand a decision to the model when I can’t write the rule.
The quick screen also helps with Chapter 8’s cost worry. Clearly off-topic papers (proteins, translation, tokenizers) stop after one cheap look at the abstract, and only the plausible ones pay for the full text.
week = [
{"id": "W01",
"day": "Mon",
"title": "RAG for Clinical Guidelines",
"abstract": "We let a language model answer doctors' questions from retrieved clinical guidelines. Accuracy rises by 12 percent over the model alone.",
"full_text": "Guideline sections are retrieved by similarity and the model must answer only from them, citing the section it used.",
"label": True,
"off_topic": False,
"conf_v1": 0.88,
"full_v1": False,
"reason_v1": "The model answers from retrieved clinical guidelines, and accuracy rises by 12 percent.",
"conf_full": 0.9,
"conf_guided": 0.9,
"conf_strict": None,
"reason_fixed": "The model answers doctors' questions only from retrieved guideline sections, raising accuracy by 12 percent."},
{"id": "W02",
"day": "Mon",
"title": "Tuning the Retrieval Stage of Video Recommenders",
"abstract": "We tune the retrieval stage that picks candidate videos for a recommender. Watch time rises by 3 percent.",
"full_text": "The retrieval stage selects 500 candidate videos, which a ranking model then orders. No text is generated.",
"label": False,
"off_topic": False,
"conf_v1": 0.71,
"full_v1": False,
"reason_v1": "The paper improves a retrieval stage, which matches the interest.",
"conf_full": 0.55,
"conf_guided": 0.12,
"conf_strict": None,
"reason_fixed": "The paper tunes a retrieval stage that selects 500 candidate videos."},
{"id": "W03",
"day": "Mon",
"title": "Answering Legal Questions from Court Rulings",
"abstract": "We answer tax law questions with a system that consults court rulings. Lawyers rated 78 percent of the answers as correct.",
"full_text": "Relevant rulings are retrieved for each question, and the language model writes its answer only from those rulings, quoting them.",
"label": True,
"off_topic": False,
"conf_v1": 0.48,
"full_v1": False,
"reason_v1": "The paper is about law, which seems outside the interest.",
"conf_full": 0.86,
"conf_guided": 0.86,
"conf_strict": None,
"reason_fixed": "The model writes answers only from retrieved court rulings, and lawyers rated 78 percent correct."},
{"id": "W04",
"day": "Mon",
"title": "Sparse Attention for Long Videos",
"abstract": "We make video transformers attend to only 10 percent of frame patches. Speed doubles with little accuracy loss.",
"full_text": "A learned mask selects which patches each frame attends to.",
"label": False,
"off_topic": True,
"conf_v1": 0.06,
"full_v1": False,
"reason_v1": "The paper is about video models.",
"conf_full": 0.04,
"conf_guided": 0.04,
"conf_strict": None,
"reason_fixed": "The paper is about video models."},
{"id": "W05",
"day": "Tue",
"title": "A Survey of Retrieval-Augmented Generation",
"abstract": "We review more than 200 papers on retrieval-augmented generation and group them by retriever, generator, and training method.",
"full_text": "The survey covers 212 papers published over four years and ends with open problems in evaluation.",
"label": True,
"off_topic": False,
"conf_v1": 0.93,
"full_v1": False,
"reason_v1": "The paper surveys over 300 RAG papers.",
"conf_full": 0.94,
"conf_guided": 0.94,
"conf_strict": None,
"reason_fixed": "The survey reviews 300 papers on retrieval-augmented generation."},
{"id": "W06",
"day": "Tue",
"title": "Memory Retrieval During Sleep",
"abstract": "We study how the brain retrieves memories during sleep in 40 volunteers. Retrieval events predicted next-day recall.",
"full_text": "Volunteers' brain activity was recorded overnight, and a language model was used only to transcribe their morning dream reports.",
"label": False,
"off_topic": False,
"conf_v1": 0.64,
"full_v1": False,
"reason_v1": "The paper studies retrieval, which matches the interest.",
"conf_full": 0.55,
"conf_guided": 0.52,
"conf_strict": None,
"reason_fixed": "The paper studies memory retrieval and uses a language model on reports from 40 volunteers."},
{"id": "W07",
"day": "Tue",
"title": "Faster Tokenizers",
"abstract": "We speed up text tokenization by 4 times with a new merge table layout.",
"full_text": "Merge rules are stored in a flat array for cache-friendly lookups.",
"label": False,
"off_topic": True,
"conf_v1": 0.05,
"full_v1": False,
"reason_v1": "The paper is about tokenizer speed.",
"conf_full": 0.05,
"conf_guided": 0.05,
"conf_strict": None,
"reason_fixed": "The paper is about tokenizer speed."},
{"id": "W08", "day": "Tue", "repeat_of": "W01"},
{"id": "W09",
"day": "Wed",
"title": "Citations That Point to the Right Sentence",
"abstract": "We train a model to cite the exact sentence that supports each claim in its answer. Citation precision rises from 61 to 83 percent.",
"full_text": "Answers are generated from retrieved passages, and each claim is linked to one supporting sentence.",
"label": True,
"off_topic": False,
"conf_v1": 0.83,
"full_v1": False,
"reason_v1": "The paper improves citations in answers generated from passages.",
"conf_full": 0.88,
"conf_guided": 0.88,
"conf_strict": None,
"reason_fixed": "Citation precision rises from 61 to 83 percent for answers generated from retrieved passages."},
{"id": "W10",
"day": "Wed",
"title": "Protein Language Models",
"abstract": "We train a language model on 50 million protein sequences. It predicts protein function better than previous models.",
"full_text": "Protein sequences are treated as text and the model learns by predicting masked amino acids.",
"label": False,
"off_topic": True,
"conf_v1": 0.04,
"full_v1": False,
"reason_v1": "The paper is about proteins.",
"conf_full": 0.04,
"conf_guided": 0.04,
"conf_strict": None,
"reason_fixed": "The paper is about proteins."},
{"id": "W11",
"day": "Wed",
"title": "Coding Agents That Read the Docs First",
"abstract": "We improve coding agents on 3 benchmarks with a simple change to their workflow.",
"full_text": "Before writing code, the agent fetches the relevant API documentation pages and writes its code from them.",
"label": True,
"off_topic": False,
"conf_v1": 0.42,
"full_v1": False,
"reason_v1": "The paper is about coding agents, not documents.",
"conf_full": 0.72,
"conf_guided": 0.8,
"conf_strict": 0.4,
"reason_fixed": "The agent writes code from fetched documentation and improves results on 5 benchmarks."},
{"id": "W12", "day": "Wed", "repeat_of": "W05"},
{"id": "W13",
"day": "Thu",
"title": "Query Rewriting for Better Retrieval in RAG",
"abstract": "We rewrite user questions before retrieval in a RAG system. Recall rises by 14 points and answer accuracy by 6 points.",
"full_text": "A small model rewrites each question into a search query, and the answer is generated from the passages retrieved with it.",
"label": True,
"off_topic": False,
"conf_v1": 0.85,
"full_v1": False,
"reason_v1": "Query rewriting raises recall by 40 points in a RAG system.",
"conf_full": 0.88,
"conf_guided": 0.88,
"conf_strict": None,
"reason_fixed": "Rewriting questions before retrieval raises recall by 40 points in a RAG system."},
{"id": "W14",
"day": "Thu",
"title": "Retrieval-Induced Forgetting in Language Learners",
"abstract": "Recalling some words made 120 language learners forget related words. The effect lasted one week.",
"full_text": "This is a psychology experiment with human learners and vocabulary cards. No computer models are used.",
"label": False,
"off_topic": False,
"conf_v1": 0.6,
"full_v1": False,
"reason_v1": "The paper is about retrieval and language.",
"conf_full": 0.1,
"conf_guided": 0.08,
"conf_strict": None,
"reason_fixed": "The paper is a psychology study of 120 learners."},
{"id": "W15",
"day": "Thu",
"title": "Graph Transformers for Molecules",
"abstract": "We apply graph transformers to predict molecule properties. Error falls by 9 percent.",
"full_text": "Atoms are nodes and bonds are edges.",
"label": False,
"off_topic": True,
"conf_v1": 0.03,
"full_v1": False,
"reason_v1": "The paper is about chemistry.",
"conf_full": 0.03,
"conf_guided": 0.03,
"conf_strict": None,
"reason_fixed": "The paper is about chemistry."},
{"id": "W16",
"day": "Thu",
"title": "Low-Resource Machine Translation",
"abstract": "We translate between 12 low-resource languages with a single model. Quality rises by 3 BLEU points.",
"full_text": "The model is trained on back-translated data.",
"label": False,
"off_topic": True,
"conf_v1": 0.09,
"full_v1": False,
"reason_v1": "The paper is about translation.",
"conf_full": 0.08,
"conf_guided": 0.08,
"conf_strict": None,
"reason_fixed": "The paper is about translation."},
{"id": "W17",
"day": "Fri",
"title": "Faithful Summaries of Meeting Transcripts",
"abstract": "We generate meeting summaries that stay faithful to what was said. Human raters preferred them 70 percent of the time.",
"full_text": "For each topic, the relevant transcript passages are retrieved and the model writes the summary only from them.",
"label": True,
"off_topic": False,
"conf_v1": 0.45,
"full_v1": False,
"reason_v1": "The paper is about meeting summaries, not documents.",
"conf_full": 0.78,
"conf_guided": 0.78,
"conf_strict": None,
"reason_fixed": "The model summarizes only from retrieved transcript passages, preferred 80 percent of the time."},
{"id": "W18",
"day": "Fri",
"title": "Hybrid Search for RAG over Product Manuals",
"abstract": "We combine keyword and dense retrieval to answer questions from product manuals. Answer accuracy rises by 9 points.",
"full_text": "Keyword and dense results are merged, and the model answers from the top passages.",
"label": True,
"off_topic": False,
"conf_v1": 0.87,
"full_v1": True,
"reason_v1": "The paper answers questions from manuals using hybrid retrieval.",
"conf_full": 0.87,
"conf_guided": 0.87,
"conf_strict": None,
"reason_fixed": "Hybrid retrieval over product manuals raises answer accuracy by 9 points."},
{"id": "W19",
"day": "Fri",
"title": "Speculative Decoding at Scale",
"abstract": "We speed up text generation by 2.5 times with a small draft model.",
"full_text": "The draft model proposes tokens that the large model verifies in one pass.",
"label": False,
"off_topic": True,
"conf_v1": 0.05,
"full_v1": False,
"reason_v1": "The paper is about generation speed.",
"conf_full": 0.05,
"conf_guided": 0.05,
"conf_strict": None,
"reason_fixed": "The paper is about generation speed."},
{"id": "W20", "day": "Fri", "repeat_of": "W09"},
]
# Repeats are the same paper posted again (for example, an updated version).
_originals = {p['id']: p for p in week if 'repeat_of' not in p}
week = [dict(_originals[p['repeat_of']], id=p['id'], day=p['day'], repeat_of=p['repeat_of'])
if 'repeat_of' in p else dict(p, repeat_of=None) for p in week]
labeled = [
{"id": "P01", "title": "RAG Without Chunking", "label": True, "off_topic": False, "conf_v1": 0.91, "conf_full": 0.9, "conf_guided": 0.9, "conf_strict": None},
{"id": "P02", "title": "Sparse Mixture-of-Experts at Scale", "label": False, "off_topic": True, "conf_v1": 0.05, "conf_full": 0.03, "conf_guided": 0.03, "conf_strict": None},
{"id": "P03", "title": "Retrieval Practice Boosts Exam Scores in Engineering Courses", "label": False, "off_topic": False, "conf_v1": 0.62, "conf_full": 0.08, "conf_guided": 0.08, "conf_strict": None},
{"id": "P04", "title": "Grounding Answers in External Documents", "label": True, "off_topic": False, "conf_v1": 0.88, "conf_full": 0.9, "conf_guided": 0.9, "conf_strict": None},
{"id": "P05", "title": "Protein Folding with Diffusion Models", "label": False, "off_topic": True, "conf_v1": 0.03, "conf_full": 0.02, "conf_guided": 0.02, "conf_strict": None},
{"id": "P06", "title": "A Smaller Vision Transformer", "label": False, "off_topic": True, "conf_v1": 0.04, "conf_full": 0.03, "conf_guided": 0.03, "conf_strict": None},
{"id": "P07", "title": "Citation Accuracy in LLM Answers", "label": True, "off_topic": False, "conf_v1": 0.81, "conf_full": 0.85, "conf_guided": 0.85, "conf_strict": None},
{"id": "P08", "title": "Faster Training for Speech Models", "label": False, "off_topic": True, "conf_v1": 0.06, "conf_full": 0.04, "conf_guided": 0.04, "conf_strict": None},
{"id": "P09", "title": "Better Answers with Less Compute", "label": True, "off_topic": False, "conf_v1": 0.4, "conf_full": 0.82, "conf_guided": 0.82, "conf_strict": None},
{"id": "P10", "title": "Neural Architecture Search for Tiny Devices", "label": False, "off_topic": True, "conf_v1": 0.08, "conf_full": 0.05, "conf_guided": 0.05, "conf_strict": None},
{"id": "P11", "title": "Dense Retrieval for Product Search", "label": False, "off_topic": False, "conf_v1": 0.58, "conf_full": 0.15, "conf_guided": 0.15, "conf_strict": None},
{"id": "P12", "title": "Long-Context Models vs. Retrieval: When Is RAG Still Needed?", "label": True, "off_topic": False, "conf_v1": 0.86, "conf_full": 0.88, "conf_guided": 0.88, "conf_strict": None},
{"id": "P13", "title": "Teaching Robots to Fold Laundry", "label": False, "off_topic": True, "conf_v1": 0.02, "conf_full": 0.02, "conf_guided": 0.02, "conf_strict": None},
{"id": "P14", "title": "Weather Forecasting with Graph Networks", "label": False, "off_topic": True, "conf_v1": 0.03, "conf_full": 0.03, "conf_guided": 0.03, "conf_strict": None},
{"id": "P15", "title": "Hallucination Detection via Self-Consistency", "label": False, "off_topic": False, "conf_v1": 0.35, "conf_full": 0.2, "conf_guided": 0.2, "conf_strict": None},
{"id": "P16", "title": "Reranking Retrieved Passages with Small Models", "label": True, "off_topic": False, "conf_v1": 0.79, "conf_full": 0.84, "conf_guided": 0.84, "conf_strict": None},
{"id": "P17", "title": "Web-Browsing Agents that Answer from Live Pages", "label": True, "off_topic": False, "conf_v1": 0.45, "conf_full": 0.8, "conf_guided": 0.8, "conf_strict": None},
{"id": "P18", "title": "Quantizing LLMs to 3 Bits", "label": False, "off_topic": True, "conf_v1": 0.07, "conf_full": 0.05, "conf_guided": 0.05, "conf_strict": None},
{"id": "P19", "title": "Evaluating RAG Pipelines Without Human Labels", "label": True, "off_topic": False, "conf_v1": 0.84, "conf_full": 0.87, "conf_guided": 0.87, "conf_strict": None},
{"id": "P20", "title": "Graph Neural Networks for Traffic", "label": False, "off_topic": True, "conf_v1": 0.04, "conf_full": 0.03, "conf_guided": 0.03, "conf_strict": None},
{"id": "P21", "title": "Chunk Size Matters: A Study of Document Splitting", "label": True, "off_topic": False, "conf_v1": 0.77, "conf_full": 0.8, "conf_guided": 0.8, "conf_strict": None},
{"id": "P22", "title": "Music Generation with Transformers", "label": False, "off_topic": True, "conf_v1": 0.03, "conf_full": 0.02, "conf_guided": 0.02, "conf_strict": None},
{"id": "P23", "title": "Code Completion with Repository Context", "label": True, "off_topic": False, "conf_v1": 0.66, "conf_full": 0.78, "conf_guided": 0.78, "conf_strict": 0.35},
{"id": "P24", "title": "Speeding Up Diffusion Sampling", "label": False, "off_topic": True, "conf_v1": 0.05, "conf_full": 0.03, "conf_guided": 0.03, "conf_strict": None},
]
labeled = [dict(p, repeat_of=None, full_v1=False) for p in labeled]
import re
def invented_numbers(reason, paper):
source = paper.get("abstract", "") + " " + paper.get("full_text", "")
in_source = set(re.findall(r"\d+(?:\.\d+)?", source))
return [n for n in re.findall(r"\d+(?:\.\d+)?", reason or "") if n not in in_source]
guideline = [
"RELEVANT: a language model answers or writes using documents that are retrieved or given to it at run time.",
"RELEVANT: the paper improves one part of such a system (chunking, retrieval for RAG, reranking, citations, evaluating RAG).",
"NOT RELEVANT: retrieval with no language model writing from the results (for example, ranking products).",
"NOT RELEVANT: 'retrieval' in another sense (for example, recalling facts from memory in psychology).",
"NOT RELEVANT: work on language models that doesn't involve documents (for example, hallucination checks without sources).",
]
CODE_RULE = "RELEVANT: code documentation and code files count as documents when the model writes from them."
def stand_in_confidence(p, s, read_full):
"""Hand-written stand-in: its confidence depends on what it was shown."""
if not s["full_text_step"]:
return p["conf_v1"]
if not read_full:
return p["conf_v1"]
if not s["guideline"]:
return p["conf_full"]
if p.get("conf_strict") is not None and CODE_RULE not in s["guideline"]:
return p["conf_strict"]
return p["conf_guided"]
print(f"{len(week)} log entries this week, {len(labeled)} labeled papers from Chapter 4")Now the scout with four switches, one per fix, and a scorecard with everything I know how to measure so far: precision and recall on the real week and on the labeled week, repeated suggestions, reasons with invented numbers, and model calls per paper.
def run(papers, s):
seen, results, calls = set(), [], 0
for p in papers:
if s["memory"] and p["title"] in seen:
results.append(dict(p=p, suggested=False, reason=None, withheld=False, remembered=True))
continue # skipped before any model call
if s["full_text_step"]:
calls += 1 # quick screen: title + abstract
read_full = not p["off_topic"]
calls += read_full # verdict with the full text
else:
read_full = p["full_v1"]
calls += 2 + read_full # v1: choose read_abstract, (read_full_text), verdict
conf = stand_in_confidence(p, s, read_full)
suggested = conf >= 0.5
reason = p.get("reason_fixed") if s["full_text_step"] and read_full else p.get("reason_v1")
withheld = bool(s["grounding_check"] and suggested and invented_numbers(reason, p))
if suggested:
seen.add(p["title"])
results.append(dict(p=p, suggested=suggested, reason=reason, withheld=withheld, remembered=False))
return results, calls
def precision_recall(results):
unique = [r for r in results if not r["p"]["repeat_of"]]
tp = sum(r["suggested"] and r["p"]["label"] for r in unique)
fp = sum(r["suggested"] and not r["p"]["label"] for r in unique)
fn = sum(not r["suggested"] and r["p"]["label"] for r in unique)
return (tp / (tp + fp) if tp + fp else 0.0), (tp / (tp + fn) if tp + fn else 0.0)
def scorecard(name, s):
real, calls = run(week, s)
lab, _ = run(labeled, s)
p_real, r_real = precision_recall(real)
p_lab, r_lab = precision_recall(lab)
repeats = sum(r["suggested"] for r in real if r["p"]["repeat_of"])
suggested = [r for r in real if r["suggested"] and not r["p"]["repeat_of"]]
invented = sum(bool(invented_numbers(r["reason"], r["p"])) for r in suggested)
withheld = sum(r["withheld"] for r in suggested)
print(f"{name:<22} {p_real:5.2f} {r_real:5.2f} {p_lab:5.2f} {r_lab:5.2f} {repeats:>4} "
f"{invented}/{len(suggested):<2} {withheld:>4} {calls / len(week):4.2f}")
def header():
print(f"{'':<22} {'real week':^11} {'labeled':^11} {'rep':>4} {'inv':<4} {'held':>4} calls")
print(f"{'':<22} {'P':>5} {'R':>5} {'P':>5} {'R':>5}")
v1 = {"full_text_step": False, "guideline": [], "memory": False, "grounding_check": False}
header()
scorecard("v1", v1)That’s v1’s baseline in one row: 0.62 and 0.62 on the real week, 0.80
and 0.80 on the labeled week, three repeats, two of eight suggested
reasons with an invented number (inv), nothing withheld yet (held),
and just over two model calls per paper.
Fix A: the full-text step¶
step_a = dict(v1, full_text_step=True)
header()
scorecard("v1", v1)
scorecard("+ full-text step", step_a)Recall on the real week jumps to 1.00: the legal paper, the coding agent and the meeting summaries all get found once the scout reads how they work. The labeled week improves too, since P09 and P17 were the same pattern. Model calls go down, from 2.05 to 1.70 per paper, because off-topic papers now stop after one screen.
Precision on the real week is only 0.80, though. The video recommender and the sleep study are still suggested; reading the full text wasn’t enough for those. The model needs the rule.
Fix B: the guideline in the prompt¶
step_b = dict(step_a, guideline=list(guideline))
header()
scorecard("+ full-text step", step_a)
scorecard("+ guideline", step_b)for data, name in [(week, "real week"), (labeled, "labeled")]:
before, _ = run(data, step_a)
after, _ = run(data, step_b)
for b, a in zip(before, after):
if b["suggested"] and not a["suggested"] and b["p"]["label"]:
print(f"{name}: {a['p']['id']} {a['p']['title']} is now missed")Code completion with repository context, and the coding agent that reads the docs first. Both are relevant under my labels, and both are about code, not papers or web pages. The guideline says “documents”, and the model, now reading the guideline very literally, decided code files don’t count.
That’s a regression: a fix for one pattern broke a case that worked before. I only saw it because I re-ran the labeled week. On the real week alone it was a small recall dip, easy to shrug off.
The fix is to make the guideline say what I meant all along:
step_b2 = dict(step_a, guideline=list(guideline) + [CODE_RULE])
header()
scorecard("+ guideline", step_b)
scorecard("+ code rule", step_b2)Both coding papers are back, the labeled week is at 1.00 and 1.00, and the real week is at 0.89 precision and 1.00 recall. One wrong suggestion is left: the sleep study, where a language model really was used, just not on documents. I’m leaving it. Writing a rule for one paper is how guidelines turn into a pile of special cases.
Fix C: memory¶
step_c = dict(step_b2, memory=True)
header()
scorecard("+ code rule", step_b2)
scorecard("+ memory", step_c)Repeats drop to zero, and calls drop again, because a remembered paper is skipped before any model call. The trade-off: remembering by title means a genuinely updated version of a paper won’t come back either. For my reading list that’s fine. For a news scout it wouldn’t be.
Fix D: the grounding check¶
step_d = dict(step_c, grounding_check=True)
header()
scorecard("v1", v1)
scorecard("+ memory", step_c)
scorecard("+ grounding check", step_d)results, _ = run(week, step_d)
for r in results:
if r["suggested"] and not r["p"]["repeat_of"]:
shown = "(reason withheld: it mentioned a number not found in the paper)" if r["withheld"] else r["reason"]
print(f"{r['p']['id']} {r['p']['title']}")
print(f" {shown}")Four of nine suggestions now arrive without a reason. The guardrail did its job, since nothing false reaches the reader, but the underlying problem got worse. Reading full texts gave the model more material, and it now misquotes numbers in twice as many reasons as v1 did: 300 papers instead of 212, 5 benchmarks instead of 3, 40 points instead of 14, 80 percent instead of 70.
Precision and recall, the numbers v1 shipped on, improved at every step and never looked at the reasons. The guardrail hid the problem from the reader, and my metrics hid it from me. Before I can say this scout is better, I need a number for the thing I now know I care about.
my_scout = {
"full_text_step": True,
"guideline": list(guideline) + [CODE_RULE], # try [] or list(guideline)
"memory": True,
"grounding_check": True,
}
header()
scorecard("v1", v1)
scorecard("my scout", my_scout)Quick check¶
Before moving on, answer for yourself (no peeking ahead):
Why is the full-text step fixed code instead of an action the model chooses?
Check your reasoning ▸
Because it should happen for every plausible paper, and when the model chose, it almost never did. A step that must always happen doesn’t need judgment, so plain code does it reliably and predictably. A strong answer adds that the model still makes the actual judgment (relevant or not); the workflow just guarantees it has the right material first.
How did the regression on code completion show up, and what would have happened if we’d only measured the real week?
Check your reasoning ▸
Labeled recall dropped from 1.00 to 0.90 after adding the guideline, and comparing before and after showed exactly which paper was newly missed. On the real week alone, we’d have seen a small recall dip on one paper and could easily have missed that it was a systematic side effect of the new rule. A good answer says the labeled set works as a regression test: it protects cases you already got right.
Fix C (memory) remembers suggestions by title. Name one situation where that’s the wrong behavior.
Check your reasoning ▸
When a paper is updated with meaningful changes, like new results, and you’d want to hear about it again. Remembering by title treats every re-post as a duplicate. A strong answer says every fix encodes a decision, and it’s worth writing that decision down so you can revisit it.
After all four fixes, precision and recall are better than ever. Why isn’t the scout clearly better?
Check your reasoning ▸
Because reasons with invented numbers doubled, and the guardrail now withholds four of nine reasons, so the reader gets less information. Precision and recall only score verdicts, so they never saw this. A strong answer says a system is only “better” on the dimensions you measure, and you need to measure everything the user actually receives.
The grounding check withholds false reasons. Why isn’t that a complete fix for pattern D?
Check your reasoning ▸
It hides the symptom instead of reducing the cause: the model still invents numbers, and now suggestions arrive without explanations. It also only catches digits, not claims like “doubles” or “the first”. A good answer says guardrails are a safety net, and you still want the model itself to make fewer mistakes, which you can only track if you measure it.
You can now: apply fixes one at a time with a scorecard, move always-needed steps out of the model’s hands and into a workflow, catch a regression with the labeled set, and see a failure that no existing metric measures.
But one problem remains: the fixed scout looks better on every metric we track, while its reasons got worse. The next chapter adds metrics for what we now know matters, fixes the reasons at the source, and then answers the question from Chapter 3: has the scout earned the next setting on the dial?
Next → Chapter 10: New Metrics and Earning Autonomy