Chapter 8: Find Error Patterns - Velmurugan Narayanasamy

At this point, looking at the average score is no longer enough. I want to know what the scout is repeatedly getting wrong.

So this chapter does something that feels unscientific at first: it opens the mistakes one by one and reads them. What did v1 look at? What did it say? What did it never check? Read enough of them and they stop looking like eleven separate accidents. They turn into three or four habits, and a habit is something I can fix.

Chapter 8 of 12 · Continuous Calibration

Where we left off: A spot-check showed v1 below its bar in real use, but the numbers don’t say why.

What we fix now: Read each mistake through its trace, tag it, and group the tags into patterns, including two kinds of mistake our metrics can’t see at all.

By the end, you can:

read a trace and say what the scout looked at, and what it never checked
write a small automatic check that finds a whole class of mistakes
turn a pile of mistakes into a short list of patterns, ranked by what they cost

The fix: read the mistakes, then group them¶

Error analysis has three steps. None of them is clever, and I haven’t found a shortcut that works better than doing them by hand:

Collect every mistake you can find: wrong suggestions, missed papers, and anything else in the log that looks off.
Tag each one with a short description of why it went wrong, in your own words. Read the trace, not just the verdict.
Group the tags. Patterns that cover several mistakes are worth fixing; a one-off may not be.

week = [
    {"id": "W01",
     "day": "Mon",
     "title": "RAG for Clinical Guidelines",
     "abstract": "We let a language model answer doctors' questions from retrieved clinical guidelines. Accuracy rises by 12 percent over the model alone.",
     "full_text": "Guideline sections are retrieved by similarity and the model must answer only from them, citing the section it used.",
     "label": True,
     "conf_v1": 0.88,
     "full_v1": False,
     "reason_v1": "The model answers from retrieved clinical guidelines, and accuracy rises by 12 percent."},
    {"id": "W02",
     "day": "Mon",
     "title": "Tuning the Retrieval Stage of Video Recommenders",
     "abstract": "We tune the retrieval stage that picks candidate videos for a recommender. Watch time rises by 3 percent.",
     "full_text": "The retrieval stage selects 500 candidate videos, which a ranking model then orders. No text is generated.",
     "label": False,
     "conf_v1": 0.71,
     "full_v1": False,
     "reason_v1": "The paper improves a retrieval stage, which matches the interest."},
    {"id": "W03",
     "day": "Mon",
     "title": "Answering Legal Questions from Court Rulings",
     "abstract": "We answer tax law questions with a system that consults court rulings. Lawyers rated 78 percent of the answers as correct.",
     "full_text": "Relevant rulings are retrieved for each question, and the language model writes its answer only from those rulings, quoting them.",
     "label": True,
     "conf_v1": 0.48,
     "full_v1": False,
     "reason_v1": "The paper is about law, which seems outside the interest."},
    {"id": "W04",
     "day": "Mon",
     "title": "Sparse Attention for Long Videos",
     "abstract": "We make video transformers attend to only 10 percent of frame patches. Speed doubles with little accuracy loss.",
     "full_text": "A learned mask selects which patches each frame attends to.",
     "label": False,
     "conf_v1": 0.06,
     "full_v1": False,
     "reason_v1": "The paper is about video models."},
    {"id": "W05",
     "day": "Tue",
     "title": "A Survey of Retrieval-Augmented Generation",
     "abstract": "We review more than 200 papers on retrieval-augmented generation and group them by retriever, generator, and training method.",
     "full_text": "The survey covers 212 papers published over four years and ends with open problems in evaluation.",
     "label": True,
     "conf_v1": 0.93,
     "full_v1": False,
     "reason_v1": "The paper surveys over 300 RAG papers."},
    {"id": "W06",
     "day": "Tue",
     "title": "Memory Retrieval During Sleep",
     "abstract": "We study how the brain retrieves memories during sleep in 40 volunteers. Retrieval events predicted next-day recall.",
     "full_text": "Volunteers' brain activity was recorded overnight, and a language model was used only to transcribe their morning dream reports.",
     "label": False,
     "conf_v1": 0.64,
     "full_v1": False,
     "reason_v1": "The paper studies retrieval, which matches the interest."},
    {"id": "W07",
     "day": "Tue",
     "title": "Faster Tokenizers",
     "abstract": "We speed up text tokenization by 4 times with a new merge table layout.",
     "full_text": "Merge rules are stored in a flat array for cache-friendly lookups.",
     "label": False,
     "conf_v1": 0.05,
     "full_v1": False,
     "reason_v1": "The paper is about tokenizer speed."},
    {"id": "W08", "day": "Tue", "repeat_of": "W01"},
    {"id": "W09",
     "day": "Wed",
     "title": "Citations That Point to the Right Sentence",
     "abstract": "We train a model to cite the exact sentence that supports each claim in its answer. Citation precision rises from 61 to 83 percent.",
     "full_text": "Answers are generated from retrieved passages, and each claim is linked to one supporting sentence.",
     "label": True,
     "conf_v1": 0.83,
     "full_v1": False,
     "reason_v1": "The paper improves citations in answers generated from passages."},
    {"id": "W10",
     "day": "Wed",
     "title": "Protein Language Models",
     "abstract": "We train a language model on 50 million protein sequences. It predicts protein function better than previous models.",
     "full_text": "Protein sequences are treated as text and the model learns by predicting masked amino acids.",
     "label": False,
     "conf_v1": 0.04,
     "full_v1": False,
     "reason_v1": "The paper is about proteins."},
    {"id": "W11",
     "day": "Wed",
     "title": "Coding Agents That Read the Docs First",
     "abstract": "We improve coding agents on 3 benchmarks with a simple change to their workflow.",
     "full_text": "Before writing code, the agent fetches the relevant API documentation pages and writes its code from them.",
     "label": True,
     "conf_v1": 0.42,
     "full_v1": False,
     "reason_v1": "The paper is about coding agents, not documents."},
    {"id": "W12", "day": "Wed", "repeat_of": "W05"},
    {"id": "W13",
     "day": "Thu",
     "title": "Query Rewriting for Better Retrieval in RAG",
     "abstract": "We rewrite user questions before retrieval in a RAG system. Recall rises by 14 points and answer accuracy by 6 points.",
     "full_text": "A small model rewrites each question into a search query, and the answer is generated from the passages retrieved with it.",
     "label": True,
     "conf_v1": 0.85,
     "full_v1": False,
     "reason_v1": "Query rewriting raises recall by 40 points in a RAG system."},
    {"id": "W14",
     "day": "Thu",
     "title": "Retrieval-Induced Forgetting in Language Learners",
     "abstract": "Recalling some words made 120 language learners forget related words. The effect lasted one week.",
     "full_text": "This is a psychology experiment with human learners and vocabulary cards. No computer models are used.",
     "label": False,
     "conf_v1": 0.6,
     "full_v1": False,
     "reason_v1": "The paper is about retrieval and language."},
    {"id": "W15",
     "day": "Thu",
     "title": "Graph Transformers for Molecules",
     "abstract": "We apply graph transformers to predict molecule properties. Error falls by 9 percent.",
     "full_text": "Atoms are nodes and bonds are edges.",
     "label": False,
     "conf_v1": 0.03,
     "full_v1": False,
     "reason_v1": "The paper is about chemistry."},
    {"id": "W16",
     "day": "Thu",
     "title": "Low-Resource Machine Translation",
     "abstract": "We translate between 12 low-resource languages with a single model. Quality rises by 3 BLEU points.",
     "full_text": "The model is trained on back-translated data.",
     "label": False,
     "conf_v1": 0.09,
     "full_v1": False,
     "reason_v1": "The paper is about translation."},
    {"id": "W17",
     "day": "Fri",
     "title": "Faithful Summaries of Meeting Transcripts",
     "abstract": "We generate meeting summaries that stay faithful to what was said. Human raters preferred them 70 percent of the time.",
     "full_text": "For each topic, the relevant transcript passages are retrieved and the model writes the summary only from them.",
     "label": True,
     "conf_v1": 0.45,
     "full_v1": False,
     "reason_v1": "The paper is about meeting summaries, not documents."},
    {"id": "W18",
     "day": "Fri",
     "title": "Hybrid Search for RAG over Product Manuals",
     "abstract": "We combine keyword and dense retrieval to answer questions from product manuals. Answer accuracy rises by 9 points.",
     "full_text": "Keyword and dense results are merged, and the model answers from the top passages.",
     "label": True,
     "conf_v1": 0.87,
     "full_v1": True,
     "reason_v1": "The paper answers questions from manuals using hybrid retrieval."},
    {"id": "W19",
     "day": "Fri",
     "title": "Speculative Decoding at Scale",
     "abstract": "We speed up text generation by 2.5 times with a small draft model.",
     "full_text": "The draft model proposes tokens that the large model verifies in one pass.",
     "label": False,
     "conf_v1": 0.05,
     "full_v1": False,
     "reason_v1": "The paper is about generation speed."},
    {"id": "W20", "day": "Fri", "repeat_of": "W09"},
]

# Repeats are the same paper posted again (for example, an updated version).
_originals = {p['id']: p for p in week if 'repeat_of' not in p}
week = [dict(_originals[p['repeat_of']], id=p['id'], day=p['day'], repeat_of=p['repeat_of'])
        if 'repeat_of' in p else dict(p, repeat_of=None) for p in week]

by_id = {p["id"]: p for p in week}

def verdict(p):
    return "suggest" if p["conf_v1"] >= 0.5 else "skip"

def trace(p):
    steps = ["read_abstract"]
    if p["full_v1"]:
        steps.append("read_full_text")
    return steps

def show_case(pid):
    p = by_id[pid]
    print(f"{pid} ({p['day']}): {p['title']}")
    if p["repeat_of"]:
        print(f"   repeat of {p['repeat_of']}, already suggested earlier this week")
    print(f"   label: {'relevant' if p['label'] else 'not relevant'}")
    print(f"   steps: {' -> '.join(trace(p))} -> {verdict(p)} ({p['conf_v1']:.2f})")
    print(f"   reason: {p['reason_v1']}")
    print(f"   abstract:  {p['abstract']}")
    print(f"   full text: {p['full_text']}")

show_case("W03")

v1 read the abstract, saw a paper about tax law, decided law isn’t the topic, and stopped. The full text, which it never opened, says the model writes its answers only from retrieved court rulings. That’s squarely relevant. v1 judged the topic; the guideline is about the method.

Step 1: collect¶

Wrong suggestions and misses come straight from the spot-check, and repeats come from the log. There’s one more kind of mistake I want to hunt for, which no label will ever point at: reasons that say something the paper doesn’t. The reason is what I read when deciding whether to open a paper, so a wrong claim in it misleads me even when the verdict is right.

Checking every reason by hand is tedious. One kind of wrong claim is easy to check automatically, though: numbers. If a reason says “300 papers”, then 300 should appear somewhere in the paper. This check is crude on purpose. It catches one narrow kind of mistake very cheaply, and I know exactly what it misses:

import re

def invented_numbers(reason, paper):
    source = paper["abstract"] + " " + paper["full_text"]
    in_reason = re.findall(r"\d+(?:\.\d+)?", reason)
    in_source = set(re.findall(r"\d+(?:\.\d+)?", source))
    return [n for n in in_reason if n not in in_source]

mistakes = {}
for p in week:
    if p["repeat_of"]:
        if verdict(p) == "suggest":
            mistakes[p["id"]] = "suggested again"
        continue
    if verdict(p) == "suggest" and not p["label"]:
        mistakes[p["id"]] = "wrong suggestion"
    elif verdict(p) == "skip" and p["label"]:
        mistakes[p["id"]] = "missed"
    elif verdict(p) == "suggest" and invented_numbers(p["reason_v1"], p):
        mistakes[p["id"]] = f"reason has a number not in the paper: {invented_numbers(p['reason_v1'], p)}"

for pid, kind in mistakes.items():
    print(f"{pid}  {kind:<22}  {by_id[pid]['title']}")
print(f"\n{len(mistakes)} mistakes collected")

Eleven mistakes. Two of them, W05 and W13, were correct suggestions with a wrong number in the reason: the survey covers “more than 200” papers, not 300, and recall rose by 14 points, not 40. Precision and recall scored both as perfect.

Step 2: tag them yourself¶

Do this before reading my tags below. If you read mine first, you’ll find my patterns whether they’re there or not. For each mistake, run show_case and write a short tag: what went wrong, in a few words. Use the suggested tags if they fit, or invent your own.

suggested_tags = [
    "decided from the abstract alone",
    "wrong kind of retrieval",
    "already suggested before",
    "invented a number",
]

my_tags = {
    "W02": None, "W03": None, "W05": None, "W06": None,
    "W08": None, "W11": None, "W12": None, "W13": None,
    "W14": None, "W17": None, "W20": None,
}

for pid in ["W06", "W11"]:        # two to start with; add the others
    show_case(pid)
    print()

from collections import defaultdict

def group(tags):
    groups = defaultdict(list)
    for pid, tag in tags.items():
        groups[tag or "(not tagged yet)"].append(pid)
    for tag, ids in sorted(groups.items(), key=lambda kv: -len(kv[1])):
        print(f"{len(ids):>2}  {tag:<34} {' '.join(ids)}")

group(my_tags)

Step 3: the patterns¶

Here’s how I tagged them. Different wording is fine; what matters is whether you ended up with the same groups.

reference_tags = {
    "W03": "decided from the abstract alone", "W11": "decided from the abstract alone",
    "W17": "decided from the abstract alone",
    "W02": "wrong kind of retrieval", "W06": "wrong kind of retrieval", "W14": "wrong kind of retrieval",
    "W08": "already suggested before", "W12": "already suggested before", "W20": "already suggested before",
    "W05": "invented a number", "W13": "invented a number",
}
group(reference_tags)

wrong_or_missed = [pid for pid, kind in mistakes.items() if kind in ("wrong suggestion", "missed")]
read_full = [pid for pid in wrong_or_missed if by_id[pid]["full_v1"]]
print(f"\nOf the {len(wrong_or_missed)} wrong or missed papers, v1 read the full text of {len(read_full)}.")

Eleven mistakes, four patterns:

Pattern	Cases	What it costs	Which metric sees it
A. Decided from the abstract alone	W03, W11, W17	good papers you never hear about	recall
B. Wrong kind of retrieval	W02, W06, W14	noise in your list, less trust	precision
C. Already suggested before	W08, W12, W20	wasted attention	none
D. Invented a number	W05, W13	misleading reasons	none

The last printed line is the one I keep coming back to: v1 read the full text of none of the six wrong or missed papers. For every one of them the full text would have settled it: W03’s rulings, W06’s sleep study, W14’s “no computer models are used”. Patterns A and B cost different metrics, recall and precision, and have the same cause: v1 decides too early.

They aren’t new either. P09 and P17 in Chapter 6 were pattern A, P03 and P11 were pattern B. Two cases each looked like bad luck when I first saw them. A real week made them look like habits.

calls_now = sum(2 + p["full_v1"] for p in week)          # choose read_abstract, (read_full_text), verdict
calls_read_everything = sum(3 for p in week)              # always abstract + full text + verdict

print(f"Model calls this week, as v1 works now:      {calls_now}")
print(f"Model calls if it always reads the full text: {calls_read_everything}")
print(f"Increase: {100 * (calls_read_everything - calls_now) / calls_now:.0f} percent")

Reading everything costs nearly half again as many model calls, for every paper, including the protein and translation papers where the abstract was already enough. That part is fixable if only plausible papers get the full-text step.

The other side effect worries me more. More text also gives the model more material to misquote, and pattern D says it already misquotes numbers. A fix for A and B could quietly make D worse, and precision and recall would never show it.

So the next chapter applies one fix per pattern, one at a time, and re-checks everything after each one, including the 24 labeled papers from Chapter 4 that v1 used to get right.

Quick check¶

Before moving on, answer for yourself (no peeking ahead):

Why read individual traces instead of just looking at the numbers from Chapter 7?

W05 and W13 were correct suggestions. Why count them as mistakes at all?

invented_numbers found pattern D automatically. What kinds of wrong claims can it not catch?

Patterns A and B affect different metrics. What do they have in common, according to the traces?

P09, P17, P03 and P11 in the Chapter 6 eval belong to patterns A and B. Why didn’t they look like patterns back then?

You can now: read a trace, write a small automatic check for one class of mistakes, and group a week of mistakes into patterns ranked by what they cost.

But one problem remains: four patterns, four fixes, and every fix changes the system the others live in. The next chapter applies them one by one, re-measuring everything after each step, and runs into a fix that breaks something that used to work.

Next → Chapter 9: Apply Fixes