Chapter 11: The Capstone - Velmurugan Narayanasamy

This paper solves hallucination in RAG by training the generator to drop unsupported sentences.

That’s a draft I rejected. The paper reports that unsupported claims fell by 30 percent, which is good work, and “solves hallucination” is a claim it never makes.

In this course’s story, v1.2 held its numbers for a second week and got promoted to “draft”. It now writes a short summary for every suggested paper, and nothing goes out until I approve it. Drafts are a new kind of output with their own ways of being wrong, so this chapter runs the whole calibration loop again at the new level, in miniature. Then it takes on the last setting on the dial: should the scout ever publish on its own?

Chapter 11 of 12 · Capstone

Where we left off: The scout was promoted to “draft”, and none of our metrics can tell a faithful summary from an exaggerated one.

What we fix now: Run one full calibration loop at “draft”, using your approve and reject decisions as labels, then decide, with evidence, how far up the dial the scout should go.

By the end, you can:

turn human review into a free, continuous source of labels
run the whole loop yourself: measure, find patterns, fix, add checks, re-measure
explain why the right amount of autonomy depends on the action, and where automatic checks stop being enough

Your reviews are labels¶

At “suggest”, labels were extra work: I had to sit down and spot-check the log. At “draft”, I review every draft anyway, because nothing is published without my yes. So every approve or reject is a label I get for free. If I also jot down why I rejected a draft, half of the error analysis is already done.

drafts = [
    {"id": "C01", "title": "Retrieval-Augmented Answers for Customer Support",
     "abstract": "We answer support tickets from a company's help articles. Resolution rate rises by 9 percent.",
     "draft": "Answers support tickets from retrieved help articles; resolution rate rises by 9 percent.",
     "approved": True, "note": ""},
    {"id": "C02", "title": "Reducing Unsupported Claims in RAG",
     "abstract": "We train the generator to drop sentences its passages don't support. Unsupported claims fall by 30 percent.",
     "draft": "This paper solves hallucination in RAG by training the generator to drop unsupported sentences.",
     "approved": False, "note": "Overclaims: unsupported claims fall by 30 percent, hallucination isn't solved."},
    {"id": "C03", "title": "Chunking Tables for Retrieval",
     "abstract": "We split tables into row groups with their headers before indexing. Table question accuracy rises by 11 points.",
     "draft": "Splits tables into row groups that keep their headers, raising table question accuracy by 11 points.",
     "approved": True, "note": ""},
    {"id": "C04", "title": "Multilingual RAG with One Retriever",
     "abstract": "We use one retriever for questions in German, Hindi, and Spanish. Accuracy matches separate retrievers per language.",
     "draft": "One retriever serves questions in German, Hindi and Spanish, matching separate per-language retrievers.",
     "approved": True, "note": ""},
    {"id": "C05", "title": "A Survey of Evaluation Methods for RAG",
     "abstract": "We review 150 papers on evaluating RAG systems and sort their metrics into four groups.",
     "draft": "An important survey that everyone working on RAG should read.",
     "approved": False, "note": "Says nothing about what's in the paper."},
    {"id": "C06", "title": "Passage Compression Before Generation",
     "abstract": "We compress retrieved passages to a third of their length before generation. Accuracy stays within 1 point.",
     "draft": "Compresses retrieved passages to a third of their length, with accuracy within 1 point.",
     "approved": True, "note": ""},
    {"id": "C07", "title": "Retrieval for Long-Form Answers",
     "abstract": "We retrieve new passages while writing long answers instead of once at the start. Factual accuracy rises by 7 points.",
     "draft": "The first method to retrieve while writing, beating every earlier approach to long-form answers.",
     "approved": False, "note": "The paper claims neither 'first' nor 'every earlier approach'."},
    {"id": "C08", "title": "Citing Sources in Voice Assistants",
     "abstract": "We add spoken source citations to a voice assistant's answers. Users trusted cited answers more in a study with 60 people.",
     "draft": "Adds spoken citations to voice answers; in a study with 60 people, users trusted cited answers more.",
     "approved": True, "note": ""},
]

approval_rate = sum(d["approved"] for d in drafts) / len(drafts)
print(f"Approved {sum(d['approved'] for d in drafts)} of {len(drafts)} drafts: approval rate {approval_rate:.2f}\n")
for d in drafts:
    if not d["approved"]:
        print(f"{d['id']}  {d['title']}")
        print(f"   draft:    {d['draft']}")
        print(f"   abstract: {d['abstract']}")
        print(f"   your note: {d['note']}\n")

Three of eight drafts rejected. That’s a lot of reading for a scout whose job is to save me reading. The notes already point at patterns, though:

Overclaiming: C02 says “solves”, C07 says “the first” and “beating every earlier approach”. Both promise more than the paper.
Saying nothing: C05 praises the survey without a single fact from it.

Both have visible markers, so both can be checked automatically, the way Chapter 8 checked numbers.

Checks for the patterns¶

Two small checks, one per pattern:

claim_words flags words that often signal overclaiming.
has_concrete_result flags a draft that mentions none of the abstract’s numbers, when the abstract has some.

They don’t decide whether a draft is good. They decide which drafts I read first.

import re

claim_words = ["solves", "solved", "fixes", "first", "best", "every", "breakthrough", "everyone", "important"]

def check_draft(d):
    flags = []
    words = re.findall(r"[a-z']+", d["draft"].lower())
    found = [w for w in claim_words if w in words]
    if found:
        flags.append(f"claim words: {found}")
    numbers_in_abstract = set(re.findall(r"\d+", d["abstract"]))
    if numbers_in_abstract and not numbers_in_abstract & set(re.findall(r"\d+", d["draft"])):
        flags.append("no concrete result from the abstract")
    return flags

def compare_checks_with_reviews(drafts):
    agree = 0
    for d in drafts:
        flags = check_draft(d)
        flagged = bool(flags)
        agree += flagged == (not d["approved"])
        review = "approved" if d["approved"] else "REJECTED"
        print(f"{d['id']}  {review:>8}   checks: {', '.join(flags) or 'ok'}")
    print(f"\nChecks agree with your review on {agree} of {len(drafts)} drafts")

compare_checks_with_reviews(drafts)

The checks flag exactly the three drafts I rejected and none of the five I approved. On eight drafts that’s encouraging, nothing more.

The fix¶

As with grounding in Chapter 10, I’d rather fewer bad drafts get written than catch them all afterwards. The draft prompt gets two new rules:

Describe only what the paper says it did and found. Include one concrete result from the abstract. Don’t use words like “solves”, “first”, “best” or “breakthrough”.

Below are the new drafts for the same papers, and my review of them. (Still hand-written; with a real model you’d re-run the scout and review what it actually writes.)

redrafts = {
    "C02": "Trains the generator to drop sentences its passages don't support; unsupported claims fall by 30 percent.",
    "C04": "A single retriever handles any language as well as separate per-language retrievers.",
    "C05": "Reviews 150 papers on evaluating RAG and sorts their metrics into four groups.",
    "C06": "Compresses retrieved passages to a third of their length without any accuracy loss beyond 1 point.",
    "C07": "Retrieves new passages while writing long answers, raising factual accuracy by 7 points.",
}
new_reviews = {
    "C04": (False, "Says 'any language'; the paper tested German, Hindi and Spanish."),
}

drafts_v2 = []
for d in drafts:
    new = dict(d)
    if d["id"] in redrafts:
        new["draft"] = redrafts[d["id"]]
        new["approved"], new["note"] = new_reviews.get(d["id"], (True, ""))
    drafts_v2.append(new)

before = sum(d["approved"] for d in drafts) / len(drafts)
after = sum(d["approved"] for d in drafts_v2) / len(drafts_v2)
print(f"Approval rate: {before:.2f} -> {after:.2f}\n")
compare_checks_with_reviews(drafts_v2)

C04 now says a single retriever handles “any language”. The paper tested three. No number to misquote, no “solves” or “first”: just an overgeneralization in calm language. A careful reader spots it at once. A word list doesn’t.

Could I add “any” to claim_words?

claim_words.append("any")          # try "all", "always" too
compare_checks_with_reviews(drafts_v2)

Adding “any” catches C04, and also flags C06, a draft I approved, because “without any accuracy loss beyond 1 point” is a perfectly accurate sentence. Word lists hit this wall quickly: a word that signals an overclaim in one sentence is harmless in the next. Past that point, checking a draft means reading it against the paper. A person can do that. A judge model might, but then I’d need an eval for the judge, which is this whole course again, one level up.

I stopped adding words here on purpose. This is as far as cheap automatic checks go for drafts, and it decides the next question.

Should the scout ever “act”?¶

At “act”, the scout publishes without me. My review disappears, and the only safety left is the automatic checks. So my condition for “act” is strict: the checks have to catch every kind of mistake I’ve seen the scout make, and its drafts have to pass my review almost always, for weeks.

def ready_for_act(drafts, weeks_at_draft):
    approval = sum(d["approved"] for d in drafts) / len(drafts)
    missed_by_checks = [d["id"] for d in drafts if not d["approved"] and not check_draft(d)]
    reasons = []
    if approval < 0.98:
        reasons.append(f"approval is {approval:.2f}, the bar is 0.98")
    if missed_by_checks:
        reasons.append(f"checks miss rejected drafts: {missed_by_checks}")
    if weeks_at_draft < 4:
        reasons.append(f"only {weeks_at_draft} week(s) at draft, the bar is 4")
    return "PROMOTE to act" if not reasons else "STAY at draft: " + "; ".join(reasons)

claim_words[:] = [w for w in claim_words if w != "any"]   # back to the list without "any"
print(ready_for_act(drafts_v2, weeks_at_draft=1))

The scout stays at “draft”, and not because it’s bad: seven of eight drafts approved, with the patterns found and fixed in one loop. It stays because the mistakes that are left are the kind only reading catches, and “act” removes the reader.

Working through this changed how I think about the dial. I don’t need one setting for the whole scout. Some of its actions are low risk and easy to check; others aren’t. So I set autonomy per action:

autonomy_by_action = {
    "suggest papers to you":                ("act",     "you're the only reader; a bad suggestion costs seconds"),
    "skip already-suggested papers":        ("act",     "memory is code, and it's checked every week"),
    "add a paper to your private list":     ("act",     "private, and easy to undo"),
    "post 'nothing relevant today'":        ("draft",   "public, but short; a quick yes is cheap"),
    "publish a paper summary on your blog": ("draft",   "public, and its remaining mistakes need a reader"),
}
for action, (level, why) in autonomy_by_action.items():
    print(f"{level:>6}  {action:<38} {why}")

“Is the agent trustworthy?” never had a useful answer. “For this action, how far does a mistake travel before someone catches it, and how often does the scout make it?” does. Each row in that table can move up or down the dial on its own, with its own metrics, promotion rules and rollback rules.

What you built¶

Chapter	What you added to the scout
1	A think, act, observe loop instead of fixed steps
2	A model for the “think” step, and checking its replies before acting
3	Tools the model can ask for, and the autonomy dial
4	A narrow scope, a labeling guideline, and 24 labeled papers
5	v1, with its scope enforced in code and a trace for every decision
6	Precision, recall, a ship bar, and v1 shipped at “suggest”
7	Measurement on real use, with a spot-check
8	Error patterns, found by reading traces, plus an automatic number check
9	Fixes one at a time: a workflow step, a guideline, memory, a guardrail, and a caught regression
10	A grounding metric, promotion rules, and rollback rules
11	A second loop at “draft”, human reviews as labels, and autonomy per action

Under all of it is the same loop, run again and again: build narrowly, measure, watch real use, find patterns, fix, add the metric I was missing, and only then think about more autonomy. If I had to drop every technique in these chapters but one, I’d keep that loop.

What’s still missing¶

Things I left out of this course on purpose:

Judge models. Using a second model to label drafts or verdicts at scale, and evaluating that judge.
Many tools and many agents. The scout has a handful of tools and works alone. Bigger agents plan longer, call many tools and hand work to each other, which multiplies the places things go wrong. I’d only reach for that when an eval shows one agent isn’t enough.
Model updates. Week C in Chapter 10 hinted at it: the same scout can change behavior when the model behind the API changes. The metrics and rollback rules are what notice it.

Two other gaps, real papers with a real model and text that tries to instruct the model, are what Chapter 12 is about.

Quick check¶

Before moving on, answer for yourself (no peeking ahead):

Why are reviews at “draft” a better source of labels than spot-checks at “suggest”?

The two checks matched your reviews on all eight original drafts. Why don’t they replace your review?

Adding “any” to claim_words caught C04. Why not just keep it?

The scout stayed at “draft” even though seven of eight drafts were approved. What exactly kept it there?

“Autonomy belongs to actions, not to agents.” Pick one action from the table and argue for moving it one level up or down.

You can now: run the whole build-and-calibrate loop yourself, turn human review into labels, write checks for known patterns, recognize where automatic checks stop, and set autonomy action by action, with evidence.

But one problem remains: every paper, verdict and draft in this course was made up, so the scout has never met a real paper or a real model. The last chapter changes that: the newest arXiv papers, a real model with your own API key, and a scout that runs by itself every weekday morning.

Next → Chapter 12: Deploy Your Scout