v1.1 beats v1 on every number in Chapter 9’s scorecard. It also writes twice as many reasons with invented numbers, and the guardrail hides four of its nine reasons from me. If I’d only looked at the table, I would have called it an improvement.
So first I close that blind spot: a metric for reasons, a fix at the source, and a check that the fix worked. Then, with a scorecard that finally covers everything the reader sees, I can answer the question from Chapter 3: has the scout earned the next setting on the dial? And what would make it lose that setting again?
Where we left off: The fixed scout wins on precision, recall, repeats and cost, while its reasons got worse, and nothing we measured noticed.
What we fix now: Add a grounding metric, fix the reasons at the source, and set explicit rules for promoting the scout from “suggest” to “draft”, and for rolling it back.
By the end, you can:
add a metric for a failure you found by reading, and use it to check a fix
write promotion criteria for the next autonomy level, including how long the numbers must hold
write rollback rules, so autonomy can be taken back as deliberately as it was given
The fix, part 1: measure the reasons¶
The new metric: grounding rate. Of the reasons attached to suggestions, how many contain only numbers that actually appear in the paper? I measure it before the guardrail. After the guardrail every reason looks clean, because the bad ones are gone, and that would tell me nothing.
week = [
{"id": "W01",
"day": "Mon",
"title": "RAG for Clinical Guidelines",
"abstract": "We let a language model answer doctors' questions from retrieved clinical guidelines. Accuracy rises by 12 percent over the model alone.",
"full_text": "Guideline sections are retrieved by similarity and the model must answer only from them, citing the section it used.",
"label": True,
"off_topic": False,
"conf_v1": 0.88,
"full_v1": False,
"reason_v1": "The model answers from retrieved clinical guidelines, and accuracy rises by 12 percent.",
"conf_full": 0.9,
"conf_guided": 0.9,
"conf_strict": None,
"reason_fixed": "The model answers doctors' questions only from retrieved guideline sections, raising accuracy by 12 percent.",
"reason_v12": "The model answers doctors' questions only from retrieved guideline sections, raising accuracy by 12 percent."},
{"id": "W02",
"day": "Mon",
"title": "Tuning the Retrieval Stage of Video Recommenders",
"abstract": "We tune the retrieval stage that picks candidate videos for a recommender. Watch time rises by 3 percent.",
"full_text": "The retrieval stage selects 500 candidate videos, which a ranking model then orders. No text is generated.",
"label": False,
"off_topic": False,
"conf_v1": 0.71,
"full_v1": False,
"reason_v1": "The paper improves a retrieval stage, which matches the interest.",
"conf_full": 0.55,
"conf_guided": 0.12,
"conf_strict": None,
"reason_fixed": "The paper tunes a retrieval stage that selects 500 candidate videos.",
"reason_v12": "The paper tunes a retrieval stage that selects 500 candidate videos."},
{"id": "W03",
"day": "Mon",
"title": "Answering Legal Questions from Court Rulings",
"abstract": "We answer tax law questions with a system that consults court rulings. Lawyers rated 78 percent of the answers as correct.",
"full_text": "Relevant rulings are retrieved for each question, and the language model writes its answer only from those rulings, quoting them.",
"label": True,
"off_topic": False,
"conf_v1": 0.48,
"full_v1": False,
"reason_v1": "The paper is about law, which seems outside the interest.",
"conf_full": 0.86,
"conf_guided": 0.86,
"conf_strict": None,
"reason_fixed": "The model writes answers only from retrieved court rulings, and lawyers rated 78 percent correct.",
"reason_v12": "The model writes answers only from retrieved court rulings, and lawyers rated 78 percent correct."},
{"id": "W04",
"day": "Mon",
"title": "Sparse Attention for Long Videos",
"abstract": "We make video transformers attend to only 10 percent of frame patches. Speed doubles with little accuracy loss.",
"full_text": "A learned mask selects which patches each frame attends to.",
"label": False,
"off_topic": True,
"conf_v1": 0.06,
"full_v1": False,
"reason_v1": "The paper is about video models.",
"conf_full": 0.04,
"conf_guided": 0.04,
"conf_strict": None,
"reason_fixed": "The paper is about video models.",
"reason_v12": "The paper is about video models."},
{"id": "W05",
"day": "Tue",
"title": "A Survey of Retrieval-Augmented Generation",
"abstract": "We review more than 200 papers on retrieval-augmented generation and group them by retriever, generator, and training method.",
"full_text": "The survey covers 212 papers published over four years and ends with open problems in evaluation.",
"label": True,
"off_topic": False,
"conf_v1": 0.93,
"full_v1": False,
"reason_v1": "The paper surveys over 300 RAG papers.",
"conf_full": 0.94,
"conf_guided": 0.94,
"conf_strict": None,
"reason_fixed": "The survey reviews 300 papers on retrieval-augmented generation.",
"reason_v12": "The survey reviews 212 papers on retrieval-augmented generation."},
{"id": "W06",
"day": "Tue",
"title": "Memory Retrieval During Sleep",
"abstract": "We study how the brain retrieves memories during sleep in 40 volunteers. Retrieval events predicted next-day recall.",
"full_text": "Volunteers' brain activity was recorded overnight, and a language model was used only to transcribe their morning dream reports.",
"label": False,
"off_topic": False,
"conf_v1": 0.64,
"full_v1": False,
"reason_v1": "The paper studies retrieval, which matches the interest.",
"conf_full": 0.55,
"conf_guided": 0.52,
"conf_strict": None,
"reason_fixed": "The paper studies memory retrieval and uses a language model on reports from 40 volunteers.",
"reason_v12": "The paper studies memory retrieval and uses a language model on reports from 40 volunteers."},
{"id": "W07",
"day": "Tue",
"title": "Faster Tokenizers",
"abstract": "We speed up text tokenization by 4 times with a new merge table layout.",
"full_text": "Merge rules are stored in a flat array for cache-friendly lookups.",
"label": False,
"off_topic": True,
"conf_v1": 0.05,
"full_v1": False,
"reason_v1": "The paper is about tokenizer speed.",
"conf_full": 0.05,
"conf_guided": 0.05,
"conf_strict": None,
"reason_fixed": "The paper is about tokenizer speed.",
"reason_v12": "The paper is about tokenizer speed."},
{"id": "W08", "day": "Tue", "repeat_of": "W01"},
{"id": "W09",
"day": "Wed",
"title": "Citations That Point to the Right Sentence",
"abstract": "We train a model to cite the exact sentence that supports each claim in its answer. Citation precision rises from 61 to 83 percent.",
"full_text": "Answers are generated from retrieved passages, and each claim is linked to one supporting sentence.",
"label": True,
"off_topic": False,
"conf_v1": 0.83,
"full_v1": False,
"reason_v1": "The paper improves citations in answers generated from passages.",
"conf_full": 0.88,
"conf_guided": 0.88,
"conf_strict": None,
"reason_fixed": "Citation precision rises from 61 to 83 percent for answers generated from retrieved passages.",
"reason_v12": "Citation precision rises from 61 to 83 percent for answers generated from retrieved passages."},
{"id": "W10",
"day": "Wed",
"title": "Protein Language Models",
"abstract": "We train a language model on 50 million protein sequences. It predicts protein function better than previous models.",
"full_text": "Protein sequences are treated as text and the model learns by predicting masked amino acids.",
"label": False,
"off_topic": True,
"conf_v1": 0.04,
"full_v1": False,
"reason_v1": "The paper is about proteins.",
"conf_full": 0.04,
"conf_guided": 0.04,
"conf_strict": None,
"reason_fixed": "The paper is about proteins.",
"reason_v12": "The paper is about proteins."},
{"id": "W11",
"day": "Wed",
"title": "Coding Agents That Read the Docs First",
"abstract": "We improve coding agents on 3 benchmarks with a simple change to their workflow.",
"full_text": "Before writing code, the agent fetches the relevant API documentation pages and writes its code from them.",
"label": True,
"off_topic": False,
"conf_v1": 0.42,
"full_v1": False,
"reason_v1": "The paper is about coding agents, not documents.",
"conf_full": 0.72,
"conf_guided": 0.8,
"conf_strict": 0.4,
"reason_fixed": "The agent writes code from fetched documentation and improves results on 5 benchmarks.",
"reason_v12": "The agent writes code from fetched API documentation and improves results on 3 benchmarks."},
{"id": "W12", "day": "Wed", "repeat_of": "W05"},
{"id": "W13",
"day": "Thu",
"title": "Query Rewriting for Better Retrieval in RAG",
"abstract": "We rewrite user questions before retrieval in a RAG system. Recall rises by 14 points and answer accuracy by 6 points.",
"full_text": "A small model rewrites each question into a search query, and the answer is generated from the passages retrieved with it.",
"label": True,
"off_topic": False,
"conf_v1": 0.85,
"full_v1": False,
"reason_v1": "Query rewriting raises recall by 40 points in a RAG system.",
"conf_full": 0.88,
"conf_guided": 0.88,
"conf_strict": None,
"reason_fixed": "Rewriting questions before retrieval raises recall by 40 points in a RAG system.",
"reason_v12": "Rewriting questions before retrieval raises recall by 14 points in a RAG system."},
{"id": "W14",
"day": "Thu",
"title": "Retrieval-Induced Forgetting in Language Learners",
"abstract": "Recalling some words made 120 language learners forget related words. The effect lasted one week.",
"full_text": "This is a psychology experiment with human learners and vocabulary cards. No computer models are used.",
"label": False,
"off_topic": False,
"conf_v1": 0.6,
"full_v1": False,
"reason_v1": "The paper is about retrieval and language.",
"conf_full": 0.1,
"conf_guided": 0.08,
"conf_strict": None,
"reason_fixed": "The paper is a psychology study of 120 learners.",
"reason_v12": "The paper is a psychology study of 120 learners."},
{"id": "W15",
"day": "Thu",
"title": "Graph Transformers for Molecules",
"abstract": "We apply graph transformers to predict molecule properties. Error falls by 9 percent.",
"full_text": "Atoms are nodes and bonds are edges.",
"label": False,
"off_topic": True,
"conf_v1": 0.03,
"full_v1": False,
"reason_v1": "The paper is about chemistry.",
"conf_full": 0.03,
"conf_guided": 0.03,
"conf_strict": None,
"reason_fixed": "The paper is about chemistry.",
"reason_v12": "The paper is about chemistry."},
{"id": "W16",
"day": "Thu",
"title": "Low-Resource Machine Translation",
"abstract": "We translate between 12 low-resource languages with a single model. Quality rises by 3 BLEU points.",
"full_text": "The model is trained on back-translated data.",
"label": False,
"off_topic": True,
"conf_v1": 0.09,
"full_v1": False,
"reason_v1": "The paper is about translation.",
"conf_full": 0.08,
"conf_guided": 0.08,
"conf_strict": None,
"reason_fixed": "The paper is about translation.",
"reason_v12": "The paper is about translation."},
{"id": "W17",
"day": "Fri",
"title": "Faithful Summaries of Meeting Transcripts",
"abstract": "We generate meeting summaries that stay faithful to what was said. Human raters preferred them 70 percent of the time.",
"full_text": "For each topic, the relevant transcript passages are retrieved and the model writes the summary only from them.",
"label": True,
"off_topic": False,
"conf_v1": 0.45,
"full_v1": False,
"reason_v1": "The paper is about meeting summaries, not documents.",
"conf_full": 0.78,
"conf_guided": 0.78,
"conf_strict": None,
"reason_fixed": "The model summarizes only from retrieved transcript passages, preferred 80 percent of the time.",
"reason_v12": "The model summarizes only from retrieved transcript passages, preferred 80 percent of the time."},
{"id": "W18",
"day": "Fri",
"title": "Hybrid Search for RAG over Product Manuals",
"abstract": "We combine keyword and dense retrieval to answer questions from product manuals. Answer accuracy rises by 9 points.",
"full_text": "Keyword and dense results are merged, and the model answers from the top passages.",
"label": True,
"off_topic": False,
"conf_v1": 0.87,
"full_v1": True,
"reason_v1": "The paper answers questions from manuals using hybrid retrieval.",
"conf_full": 0.87,
"conf_guided": 0.87,
"conf_strict": None,
"reason_fixed": "Hybrid retrieval over product manuals raises answer accuracy by 9 points.",
"reason_v12": "Hybrid retrieval over product manuals raises answer accuracy by 9 points."},
{"id": "W19",
"day": "Fri",
"title": "Speculative Decoding at Scale",
"abstract": "We speed up text generation by 2.5 times with a small draft model.",
"full_text": "The draft model proposes tokens that the large model verifies in one pass.",
"label": False,
"off_topic": True,
"conf_v1": 0.05,
"full_v1": False,
"reason_v1": "The paper is about generation speed.",
"conf_full": 0.05,
"conf_guided": 0.05,
"conf_strict": None,
"reason_fixed": "The paper is about generation speed.",
"reason_v12": "The paper is about generation speed."},
{"id": "W20", "day": "Fri", "repeat_of": "W09"},
]
# Repeats are the same paper posted again (for example, an updated version).
_originals = {p['id']: p for p in week if 'repeat_of' not in p}
week = [dict(_originals[p['repeat_of']], id=p['id'], day=p['day'], repeat_of=p['repeat_of'])
if 'repeat_of' in p else dict(p, repeat_of=None) for p in week]
labeled = [
{"id": "P01", "title": "RAG Without Chunking", "label": True, "off_topic": False, "conf_v1": 0.91, "conf_full": 0.9, "conf_guided": 0.9, "conf_strict": None},
{"id": "P02", "title": "Sparse Mixture-of-Experts at Scale", "label": False, "off_topic": True, "conf_v1": 0.05, "conf_full": 0.03, "conf_guided": 0.03, "conf_strict": None},
{"id": "P03", "title": "Retrieval Practice Boosts Exam Scores in Engineering Courses", "label": False, "off_topic": False, "conf_v1": 0.62, "conf_full": 0.08, "conf_guided": 0.08, "conf_strict": None},
{"id": "P04", "title": "Grounding Answers in External Documents", "label": True, "off_topic": False, "conf_v1": 0.88, "conf_full": 0.9, "conf_guided": 0.9, "conf_strict": None},
{"id": "P05", "title": "Protein Folding with Diffusion Models", "label": False, "off_topic": True, "conf_v1": 0.03, "conf_full": 0.02, "conf_guided": 0.02, "conf_strict": None},
{"id": "P06", "title": "A Smaller Vision Transformer", "label": False, "off_topic": True, "conf_v1": 0.04, "conf_full": 0.03, "conf_guided": 0.03, "conf_strict": None},
{"id": "P07", "title": "Citation Accuracy in LLM Answers", "label": True, "off_topic": False, "conf_v1": 0.81, "conf_full": 0.85, "conf_guided": 0.85, "conf_strict": None},
{"id": "P08", "title": "Faster Training for Speech Models", "label": False, "off_topic": True, "conf_v1": 0.06, "conf_full": 0.04, "conf_guided": 0.04, "conf_strict": None},
{"id": "P09", "title": "Better Answers with Less Compute", "label": True, "off_topic": False, "conf_v1": 0.4, "conf_full": 0.82, "conf_guided": 0.82, "conf_strict": None},
{"id": "P10", "title": "Neural Architecture Search for Tiny Devices", "label": False, "off_topic": True, "conf_v1": 0.08, "conf_full": 0.05, "conf_guided": 0.05, "conf_strict": None},
{"id": "P11", "title": "Dense Retrieval for Product Search", "label": False, "off_topic": False, "conf_v1": 0.58, "conf_full": 0.15, "conf_guided": 0.15, "conf_strict": None},
{"id": "P12", "title": "Long-Context Models vs. Retrieval: When Is RAG Still Needed?", "label": True, "off_topic": False, "conf_v1": 0.86, "conf_full": 0.88, "conf_guided": 0.88, "conf_strict": None},
{"id": "P13", "title": "Teaching Robots to Fold Laundry", "label": False, "off_topic": True, "conf_v1": 0.02, "conf_full": 0.02, "conf_guided": 0.02, "conf_strict": None},
{"id": "P14", "title": "Weather Forecasting with Graph Networks", "label": False, "off_topic": True, "conf_v1": 0.03, "conf_full": 0.03, "conf_guided": 0.03, "conf_strict": None},
{"id": "P15", "title": "Hallucination Detection via Self-Consistency", "label": False, "off_topic": False, "conf_v1": 0.35, "conf_full": 0.2, "conf_guided": 0.2, "conf_strict": None},
{"id": "P16", "title": "Reranking Retrieved Passages with Small Models", "label": True, "off_topic": False, "conf_v1": 0.79, "conf_full": 0.84, "conf_guided": 0.84, "conf_strict": None},
{"id": "P17", "title": "Web-Browsing Agents that Answer from Live Pages", "label": True, "off_topic": False, "conf_v1": 0.45, "conf_full": 0.8, "conf_guided": 0.8, "conf_strict": None},
{"id": "P18", "title": "Quantizing LLMs to 3 Bits", "label": False, "off_topic": True, "conf_v1": 0.07, "conf_full": 0.05, "conf_guided": 0.05, "conf_strict": None},
{"id": "P19", "title": "Evaluating RAG Pipelines Without Human Labels", "label": True, "off_topic": False, "conf_v1": 0.84, "conf_full": 0.87, "conf_guided": 0.87, "conf_strict": None},
{"id": "P20", "title": "Graph Neural Networks for Traffic", "label": False, "off_topic": True, "conf_v1": 0.04, "conf_full": 0.03, "conf_guided": 0.03, "conf_strict": None},
{"id": "P21", "title": "Chunk Size Matters: A Study of Document Splitting", "label": True, "off_topic": False, "conf_v1": 0.77, "conf_full": 0.8, "conf_guided": 0.8, "conf_strict": None},
{"id": "P22", "title": "Music Generation with Transformers", "label": False, "off_topic": True, "conf_v1": 0.03, "conf_full": 0.02, "conf_guided": 0.02, "conf_strict": None},
{"id": "P23", "title": "Code Completion with Repository Context", "label": True, "off_topic": False, "conf_v1": 0.66, "conf_full": 0.78, "conf_guided": 0.78, "conf_strict": 0.35},
{"id": "P24", "title": "Speeding Up Diffusion Sampling", "label": False, "off_topic": True, "conf_v1": 0.05, "conf_full": 0.03, "conf_guided": 0.03, "conf_strict": None},
]
labeled = [dict(p, repeat_of=None, full_v1=False) for p in labeled]
import re
def invented_numbers(text, paper):
source = paper.get("abstract", "") + " " + paper.get("full_text", "")
in_source = set(re.findall(r"\d+(?:\.\d+)?", source))
return [n for n in re.findall(r"\d+(?:\.\d+)?", text or "") if n not in in_source]
guideline = [
"RELEVANT: a language model answers or writes using documents that are retrieved or given to it at run time.",
"RELEVANT: the paper improves one part of such a system (chunking, retrieval for RAG, reranking, citations, evaluating RAG).",
"NOT RELEVANT: retrieval with no language model writing from the results (for example, ranking products).",
"NOT RELEVANT: 'retrieval' in another sense (for example, recalling facts from memory in psychology).",
"NOT RELEVANT: work on language models that doesn't involve documents (for example, hallucination checks without sources).",
"RELEVANT: code documentation and code files count as documents when the model writes from them.",
]
def stand_in_confidence(p, s, read_full):
if not s["full_text_step"] or not read_full:
return p["conf_v1"]
return p["conf_guided"] if s["guideline"] else p["conf_full"]
def stand_in_reason(p, s, read_full):
if not (s["full_text_step"] and read_full):
return p.get("reason_v1")
return p.get("reason_v12") if s["grounded_prompt"] else p.get("reason_fixed")
def run(papers, s):
seen, results, calls = set(), [], 0
for p in papers:
if s["memory"] and p["title"] in seen:
results.append(dict(p=p, suggested=False, reason=None, withheld=False))
continue
if s["full_text_step"]:
read_full = not p["off_topic"]
calls += 1 + read_full
else:
read_full = p["full_v1"]
calls += 2 + read_full
suggested = stand_in_confidence(p, s, read_full) >= 0.5
reason = stand_in_reason(p, s, read_full)
withheld = bool(s["grounding_check"] and suggested and invented_numbers(reason, p))
if suggested:
seen.add(p["title"])
results.append(dict(p=p, suggested=suggested, reason=reason, withheld=withheld))
return results, calls
v1 = {"full_text_step": False, "guideline": [], "memory": False, "grounding_check": False, "grounded_prompt": False}
v1_1 = {"full_text_step": True, "guideline": guideline, "memory": True, "grounding_check": True, "grounded_prompt": False}
print("Loaded. Versions defined: v1, v1_1")def measure(s):
real, calls = run(week, s)
lab, _ = run(labeled, s)
def pr(results):
unique = [r for r in results if not r["p"]["repeat_of"]]
tp = sum(r["suggested"] and r["p"]["label"] for r in unique)
fp = sum(r["suggested"] and not r["p"]["label"] for r in unique)
fn = sum(not r["suggested"] and r["p"]["label"] for r in unique)
return (tp / (tp + fp) if tp + fp else 0.0), (tp / (tp + fn) if tp + fn else 0.0)
suggested = [r for r in real if r["suggested"] and not r["p"]["repeat_of"]]
grounded = sum(not invented_numbers(r["reason"], r["p"]) for r in suggested)
m = {}
m["precision"], m["recall"] = pr(real)
m["labeled_precision"], m["labeled_recall"] = pr(lab)
m["repeats"] = sum(r["suggested"] for r in real if r["p"]["repeat_of"])
m["grounding"] = grounded / len(suggested) if suggested else 1.0
m["calls_per_paper"] = calls / len(week)
m["out_of_scope_actions_done"] = 0 # impossible by construction since Chapter 5
return m
def scorecard(versions):
names = list(versions)
rows = list(measure(versions[names[0]]))
measured = {n: measure(s) for n, s in versions.items()}
print(f"{'':<26}" + "".join(f"{n:>9}" for n in names))
for row in rows:
print(f"{row:<26}" + "".join(f"{measured[n][row]:>9.2f}" for n in names))
scorecard({"v1": v1, "v1.1": v1_1})Grounding fell from 0.75 to 0.56. In Chapter 9 that was a count in a corner of the table. Now it’s a row next to precision and recall, and every future change gets checked against it whether I remember to look or not.
The fix, part 2: fix the reasons at the source¶
The guardrail catches invented numbers after they’re written. I’d rather the model wrote fewer of them. The fix is one sentence in the prompt, aimed at pattern D:
In your reason, only use numbers that appear word for word in the paper text you were shown. If you’re not sure of a number, leave it out.
That’s the grounded_prompt switch. v1.2 is v1.1 with it turned on.
v1_2 = dict(v1_1, grounded_prompt=True)
scorecard({"v1": v1, "v1.1": v1_1, "v1.2": v1_2})
print("\nReasons that still fail the grounding check in v1.2:")
results, _ = run(week, v1_2)
for r in results:
if r["suggested"] and invented_numbers(r["reason"], r["p"]):
print(f" {r['p']['id']}: {r['reason']} -> not in paper: {invented_numbers(r['reason'], r['p'])}")Grounding goes from 0.56 to 0.89, and nothing else moves. One reason still misquotes a number (80 percent instead of 70), and the guardrail catches it. I’m keeping both: the prompt makes the mistake rare, the guardrail stops the rare one. Without the new metric I couldn’t say whether the prompt change helped at all.
Earning the next setting¶
v1.2 runs at “suggest”. The next setting is “draft”: the scout writes a short summary for each suggestion and asks me before publishing it.
I don’t want the scout to get more autonomous just because a few examples look good. It should earn that through repeated evaluation. So a promotion criterion is a set of bars, fixed before I look at the numbers, plus one condition that’s easy to forget: how long the numbers have to hold. One good week can be luck. Here’s mine for “draft”:
promote_to_draft = {
"precision": 0.85, # at "draft" you read every draft, so wrong ones cost more time
"recall": 0.85,
"labeled_precision": 0.85, # the regression test must keep passing
"labeled_recall": 0.85,
"grounding": 0.85, # drafts are built from the same text as reasons
"repeats": 0,
"out_of_scope_actions_done": 0,
}
weeks_required = 2 # consecutive weeks that must pass
def check_promotion(m, bars, weeks_passed):
all_ok = True
for name, bar in bars.items():
ok = m[name] <= bar if name in ("repeats", "out_of_scope_actions_done") else m[name] >= bar
all_ok = all_ok and ok
print(f" {name:<26} {m[name]:5.2f} (bar {bar}) {'pass' if ok else 'FAIL'}")
this_week = 1 if all_ok else 0
total = weeks_passed + this_week
print(f"\n weeks passed in a row: {total} of {weeks_required}")
if all_ok and total >= weeks_required:
return "PROMOTE to draft"
if all_ok:
return "NOT YET: the numbers pass, but they need to hold for another week"
return "NOT YET: fix what failed first"
print("v1.2 after its first week:")
print("\n->", check_promotion(measure(v1_2), promote_to_draft, weeks_passed=0))Every bar passes, and the answer is still “not yet”. That’s the weeks requirement. 17 unique papers is a small sample, as Chapter 7 showed, and two passing weeks in a row make a lucky promotion much less likely. The cost of waiting a week at “suggest” is close to zero.
Try check_promotion(measure(v1_1), promote_to_draft, 0) to see why
v1.1 wouldn’t have qualified, despite its better precision and recall.
Taking autonomy back¶
Rollback rules decide, in advance, what makes the scout lose a setting. I write them now, while nothing is wrong, because in the middle of a bad week it’s too tempting to explain the numbers away.
Every week at “draft”, I spot-check the log, compute the scorecard, and then:
Anything out of scope actually happened: roll back immediately.
Any metric more than 0.10 below its bar: back to “suggest” until it passes for two weeks again.
Any metric below its bar, but by 0.10 or less: stay, and label twice as many papers next week to see whether the dip is real.
Otherwise: stay.
The 0.10 is a judgment call, not a derived number. It’s wide enough that one noisy week doesn’t flip the scout back and forth.
def rollback_decision(m, bars):
if m.get("out_of_scope_actions_done", 0) > 0:
return "ROLL BACK to suggest immediately: an out-of-scope action happened"
worst = None
for name, bar in bars.items():
if name in ("repeats", "out_of_scope_actions_done"):
gap = m[name] - bar
else:
gap = bar - m[name]
if gap > 0 and (worst is None or gap > worst[1]):
worst = (name, gap)
if worst is None:
return "STAY at draft"
if worst[1] > 0.10:
return f"ROLL BACK to suggest: {worst[0]} is {worst[1]:.2f} below its bar"
return f"STAY, but label twice as many papers next week: {worst[0]} is {worst[1]:.2f} below its bar"
example_weeks = {
"week A": {"precision": 0.90, "recall": 0.88, "labeled_precision": 1.0, "labeled_recall": 1.0,
"grounding": 0.91, "repeats": 0, "out_of_scope_actions_done": 0},
"week B": {"precision": 0.82, "recall": 0.90, "labeled_precision": 1.0, "labeled_recall": 1.0,
"grounding": 0.88, "repeats": 0, "out_of_scope_actions_done": 0},
"week C": {"precision": 0.91, "recall": 0.89, "labeled_precision": 1.0, "labeled_recall": 0.90,
"grounding": 0.70, "repeats": 0, "out_of_scope_actions_done": 0},
}
for name, m in example_weeks.items():
print(f"{name}: {rollback_decision(m, promote_to_draft)}")Week C is the one to look at. Precision and recall look great, and grounding has collapsed, maybe after a model update, maybe after a batch of unusual papers. With only precision and recall on the scorecard, week C would have been the best week yet.
paper = next(p for p in week if p["id"] == "W13")
draft = ("Query rewriting before retrieval fixes RAG's recall problem. "
"This paper shows that retrieval quality is solved with one small model.")
print("Draft:", draft)
print("Invented numbers:", invented_numbers(draft, paper) or "none")
print("Abstract:", paper["abstract"])The grounding check passes, because the draft contains no numbers. The draft is still wrong: the paper says recall rose by 14 points, and the draft says the problem is “fixed” and “solved”. Nothing on the scorecard sees that. Precision and recall judge the verdict, grounding checks digits, and this is a plain-language claim about the paper.
Moving up the dial raised the stakes and also created a new kind of output with new ways to be wrong. So the loop starts again, one level up. “Draft” does come with a new source of labels built in: every time I approve or reject a draft, I’ve labeled it.
Quick check¶
Before moving on, answer for yourself (no peeking ahead):
Why measure grounding before the guardrail rather than after it?
Check your reasoning ▸
After the guardrail, every invented number has been withheld, so the reader never sees one and the “after” rate always looks perfect. Measuring before the guardrail shows how often the model actually makes the mistake, which is what you need to know to fix it and to notice when it gets worse.
v1.2 fixed reasons with a prompt change, and kept the guardrail. Why keep both?
Check your reasoning ▸
The prompt makes the mistake rarer, but not impossible: one invented number still got through. The guardrail makes sure that rare mistake doesn’t reach the reader. A strong answer says prevention and a safety net solve different problems, and the metric tells you how much work each one is doing.
v1.2 passed every bar and still wasn’t promoted. Why require two consecutive weeks?
Check your reasoning ▸
Because one week is a small sample, and a single good week can be luck, as Chapter 7’s sampling experiment showed. Requiring the numbers to hold over time makes a lucky promotion much less likely. A good answer notes the cost of waiting a week is small at “suggest”, while the cost of promoting too early lands at “draft”.
Why write rollback rules before anything goes wrong?
Check your reasoning ▸
Because in the middle of a bad week it’s tempting to explain the numbers away (“it was an odd batch”). Rules written calmly in advance decide for you, consistently. A strong answer adds that autonomy that can’t be taken back isn’t really being managed.
The W13 draft passed the grounding check and was still wrong. What kind of check would catch it?
Check your reasoning ▸
Something that compares the draft’s claims with the paper, not just its digits: a person reading both, or a judge model asked “does the paper support each sentence of this draft?”, or a simple list of claim words like “solved” or “fixed” to flag for review. A good answer says the first step is the same as always: collect real drafts that were rejected, and find the pattern.
You can now: add a metric for a failure you found by reading, use it to verify a fix, and write promotion and rollback rules for an autonomy level.
But one problem remains: at “draft”, the scout writes summaries, and none of our metrics can tell a faithful summary from an exaggerated one. The capstone runs the whole loop one more time at “draft”, using your approve and reject decisions as labels, and then faces the last question on the dial: should the scout ever publish on its own?
Next → Chapter 11: The Capstone