Chapter 4: Scope and Curate Data - Velmurugan Narayanasamy

How would you know that a change to the scout made it better?

After day 6, the obvious move is to open the prompt and start fixing: “always read the full text”, “retrieval practice doesn’t count”. That might fix day 6. It might also break day 5, and I’d have no way of noticing, because I’d be judging each change by whichever example I happened to look at last.

So this chapter doesn’t touch the scout at all. It does the two things I should have done before writing any prompt: decide exactly what the first version is for, and collect examples where I already know the right answer. It’s the least exciting chapter in the course. Every chapter after it depends on it.

Chapter 4 of 12 · Continuous Development

Where we left off: The scout can act, and we have an autonomy dial, but no way to know when it’s safe to turn it up.

What we fix now: Start building on purpose: pick the narrowest useful first version, write down exactly what it may and may not do, and label a small set of papers with the right answers.

By the end, you can:

explain why the first version starts at the lowest autonomy setting
write a scope for v1, including the actions it must never take, and a guideline that decides edge cases
label papers yourself and see exactly where your judgment and the reference disagree

Two halves: build, then calibrate¶

From here on I follow the CC/CD lifecycle from Aishwarya Reganti and Kiriti Badam’s article Why your AI product needs a different development lifecycle (Lenny’s Newsletter). Chapters 4 to 9 follow its steps. The explanations and the scout are mine.

Continuous Development (Chapters 4 to 6): decide what the scout should do, build exactly that, measure it on examples with known answers, and ship it at low autonomy.
Continuous Calibration (Chapters 7 to 10): watch it on real use, find patterns in its mistakes, fix them, measure again. The scout only moves up the dial when the numbers hold.

Both halves are called continuous because a system that makes judgments keeps meeting inputs nobody planned for. You don’t finish it. At best, it gets quieter.

The fix, part 1: scope v1 narrowly¶

The narrowest version of the scout that’s still useful is the suggest setting from Chapter 3: it reads papers and suggests which ones to read, with a reason. It never publishes, never emails, never changes anything. I make every real decision.

I start this low on purpose. At “suggest” a mistake costs me a few seconds, so I can watch the scout’s mistakes happen instead of cleaning up after them. And a narrow job is easy to check: each paper gets one verdict, suggest or skip, that I can compare with the right answer.

That changes one design choice. In Chapter 3 the scout picked one paper per day. v1 gives every paper its own verdict, so every paper can be checked, including the ones it passes over. Here’s the scope as code, so the next chapter can enforce it:

scope_v1 = {
    "job": "For each paper, decide suggest or skip, with a one-sentence reason.",
    "autonomy": "suggest",
    "allowed_actions": ["read_abstract", "read_full_text", "suggest", "skip"],
    "never": ["publish anything", "email anyone", "change the reading list"],
    "success": "Suggests most of the relevant papers and few irrelevant ones, "
               "measured on labeled papers (Chapter 6).",
}

for key, value in scope_v1.items():
    print(f"{key:>16}: {value}")

The fix, part 2: a guideline for the hard cases¶

“Papers about RAG, or about making language models answer from documents” sounded clear in Chapter 2. It stops being clear the moment you label real papers. Is a paper about dense retrieval for product search relevant? It’s retrieval, but no language model writes anything. What about code completion with repository context? No documents in the usual sense, but the model writes code from files it was handed.

You can’t label those consistently without deciding, and writing the decision down. That’s a labeling guideline. Here’s ours:

guideline = [
    "RELEVANT: a language model answers or writes using documents that are retrieved or given to it at run time.",
    "RELEVANT: the paper improves one part of such a system (chunking, retrieval for RAG, reranking, citations, evaluating RAG).",
    "NOT RELEVANT: retrieval with no language model writing from the results (for example, ranking products).",
    "NOT RELEVANT: 'retrieval' in another sense (for example, recalling facts from memory in psychology).",
    "NOT RELEVANT: work on language models that doesn't involve documents (for example, hallucination checks without sources).",
]
for rule in guideline:
    print("-", rule)

I didn’t write these five rules in advance. Each one came from a paper I couldn’t decide on: I made a call, then wrote it down so I’d make the same call next time. The product-search rule came from P11, the psychology rule from P03. Writing the guideline is mostly finding out what I actually meant by my interest.

The data: one week of papers¶

Below is the dataset the rest of the course uses: 24 papers across three days. Like the papers in earlier chapters, they’re made up for this course, and I wrote several of them as traps: papers that use the word “retrieval” for something else, and relevant papers whose abstract hides the method. I kept the set to 24 so you can read every paper. A small set I know well is more useful here than 200 I’ve skimmed. Each one has a title, an abstract, a short full text, and a reference label: True if it’s relevant under the guideline.

papers = [
    {"id": "P01",
     "day": "Mon",
     "title": "RAG Without Chunking",
     "abstract": "We show that RAG systems can skip chunking by indexing whole sections with a structure-aware encoder. Answer accuracy matches chunked baselines on two QA datasets.",
     "full_text": "Each document section is embedded as one unit, retrieved by similarity, and passed to the language model as context for answering.",
     "label": True},
    {"id": "P02",
     "day": "Mon",
     "title": "Sparse Mixture-of-Experts at Scale",
     "abstract": "We route each token to 2 of 64 experts and train a 40B parameter model. Training cost falls by 30 percent.",
     "full_text": "A learned router sends each token to two expert networks, so only a small part of the model runs per token.",
     "label": False},
    {"id": "P03",
     "day": "Mon",
     "title": "Retrieval Practice Boosts Exam Scores in Engineering Courses",
     "abstract": "Students who practiced retrieving facts from memory scored 14 percent higher on final exams than students who reread their notes. We studied 300 students across four courses.",
     "full_text": "Retrieval practice means recalling information from memory, for example with flashcards, instead of rereading. No computer systems are involved.",
     "label": False},
    {"id": "P04",
     "day": "Mon",
     "title": "Grounding Answers in External Documents",
     "abstract": "We make a language model answer only from passages fetched from a document store. Unsupported claims fall by 60 percent.",
     "full_text": "Passages are fetched with a dense retriever and the model is trained to refuse when they do not contain the answer.",
     "label": True},
    {"id": "P05",
     "day": "Mon",
     "title": "Protein Folding with Diffusion Models",
     "abstract": "We generate protein structures with a diffusion model. Predicted structures match lab results on 71 percent of targets.",
     "full_text": "A diffusion model denoises random 3D coordinates into a protein backbone.",
     "label": False},
    {"id": "P06",
     "day": "Mon",
     "title": "A Smaller Vision Transformer",
     "abstract": "We prune a vision transformer to a third of its size. Accuracy drops by less than one point.",
     "full_text": "Attention heads with low importance scores are removed and the model is briefly retrained.",
     "label": False},
    {"id": "P07",
     "day": "Mon",
     "title": "Citation Accuracy in LLM Answers",
     "abstract": "We measure how often cited passages actually support the sentences that cite them. Only 64 percent of citations fully support their claims.",
     "full_text": "Answers are generated from retrieved passages with inline citations, and annotators check each citation against its passage.",
     "label": True},
    {"id": "P08",
     "day": "Mon",
     "title": "Faster Training for Speech Models",
     "abstract": "We cut speech model training time in half with a new batching scheme. Word error rate stays the same.",
     "full_text": "Audio clips of similar length are grouped into the same batch so less padding is wasted.",
     "label": False},
    {"id": "P09",
     "day": "Tue",
     "title": "Better Answers with Less Compute",
     "abstract": "We introduce a simple method that improves answer quality on two question answering benchmarks while using less compute. Code is available online.",
     "full_text": "Before answering, our method retrieves the five most similar passages from a Wikipedia index and adds them to the prompt. Because the answer comes from the passages, a much smaller model is enough.",
     "label": True},
    {"id": "P10",
     "day": "Tue",
     "title": "Neural Architecture Search for Tiny Devices",
     "abstract": "We search over 10,000 network designs to find models that fit on a microcontroller. The best design runs in 40 milliseconds.",
     "full_text": "An evolutionary search mutates network designs and keeps the ones that meet a latency budget.",
     "label": False},
    {"id": "P11",
     "day": "Tue",
     "title": "Dense Retrieval for Product Search",
     "abstract": "We train a dense retriever that ranks products for shopping queries. Click-through rate rises by 9 percent in an online test.",
     "full_text": "Queries and product descriptions are embedded in one vector space, and the top products are shown to the shopper. No language model generates text.",
     "label": False},
    {"id": "P12",
     "day": "Tue",
     "title": "Long-Context Models vs. Retrieval: When Is RAG Still Needed?",
     "abstract": "We compare feeding whole documents into long-context models with retrieving a few passages. Retrieval wins on cost and matches accuracy for most questions.",
     "full_text": "Both setups answer the same questions over the same documents, and we measure accuracy, latency and cost.",
     "label": True},
    {"id": "P13",
     "day": "Tue",
     "title": "Teaching Robots to Fold Laundry",
     "abstract": "We train a robot arm to fold shirts from 200 demonstrations. It succeeds on 8 of 10 unseen garments.",
     "full_text": "A policy network maps camera images to arm motions and is trained by imitation.",
     "label": False},
    {"id": "P14",
     "day": "Tue",
     "title": "Weather Forecasting with Graph Networks",
     "abstract": "We forecast 10 day weather on a global graph. Error falls by 12 percent against the previous best model.",
     "full_text": "The globe is split into a mesh of nodes and a graph network predicts the next state of each node.",
     "label": False},
    {"id": "P15",
     "day": "Tue",
     "title": "Hallucination Detection via Self-Consistency",
     "abstract": "We detect hallucinated answers by sampling a model several times and checking whether the samples agree. Detection accuracy reaches 81 percent.",
     "full_text": "No documents are used: the model is compared only against its own repeated answers.",
     "label": False},
    {"id": "P16",
     "day": "Tue",
     "title": "Reranking Retrieved Passages with Small Models",
     "abstract": "We show that a small cross-encoder can rerank retrieved passages before they reach the generator. RAG answer accuracy rises by 5 points.",
     "full_text": "The retriever returns 50 passages, the cross-encoder keeps the best 5, and the language model answers from those.",
     "label": True},
    {"id": "P17",
     "day": "Wed",
     "title": "Web-Browsing Agents that Answer from Live Pages",
     "abstract": "We build an agent that browses the web and answers questions. It beats a closed-book model by 18 points.",
     "full_text": "The agent searches the web, opens the top pages, and writes its answer only from the text of those pages, citing each one.",
     "label": True},
    {"id": "P18",
     "day": "Wed",
     "title": "Quantizing LLMs to 3 Bits",
     "abstract": "We compress a language model's weights to 3 bits with little loss in quality. Memory use falls by 80 percent.",
     "full_text": "Weights are grouped and each group gets its own scale factor before rounding.",
     "label": False},
    {"id": "P19",
     "day": "Wed",
     "title": "Evaluating RAG Pipelines Without Human Labels",
     "abstract": "We score RAG answers for faithfulness and relevance using a judge model instead of human labels. Scores agree with humans 87 percent of the time.",
     "full_text": "A judge model reads the question, the retrieved passages and the answer, and rates whether the answer is supported.",
     "label": True},
    {"id": "P20",
     "day": "Wed",
     "title": "Graph Neural Networks for Traffic",
     "abstract": "We predict traffic speed on city roads with a graph neural network. Error falls by 8 percent.",
     "full_text": "Road sensors are nodes and roads are edges in the graph.",
     "label": False},
    {"id": "P21",
     "day": "Wed",
     "title": "Chunk Size Matters: A Study of Document Splitting",
     "abstract": "We test chunk sizes from 128 to 2048 tokens across six RAG datasets. Mid-sized chunks work best for most questions.",
     "full_text": "Documents are split at each size, indexed, and used for retrieval-augmented answering.",
     "label": True},
    {"id": "P22",
     "day": "Wed",
     "title": "Music Generation with Transformers",
     "abstract": "We generate piano music with a transformer trained on 10,000 hours of recordings. Listeners preferred it 60 percent of the time.",
     "full_text": "Audio is converted to tokens and the transformer predicts the next token.",
     "label": False},
    {"id": "P23",
     "day": "Wed",
     "title": "Code Completion with Repository Context",
     "abstract": "We retrieve related files from the same repository and add them to the prompt of a code model. Completion accuracy rises by 11 points.",
     "full_text": "Files are ranked by similarity to the current file and the top three are placed in the prompt before completing code.",
     "label": True},
    {"id": "P24",
     "day": "Wed",
     "title": "Speeding Up Diffusion Sampling",
     "abstract": "We reduce diffusion sampling from 50 steps to 8 steps. Image quality stays close to the original.",
     "full_text": "A student model is trained to match several teacher steps at once.",
     "label": False},
]

print(f"{len(papers)} papers\n")
for p in papers:
    mark = "RELEVANT" if p["label"] else "   -    "
    print(f"{p['id']}  {p['day']}  {mark}  {p['title']}")

Ten of the 24 are relevant. Most are obvious either way; nobody needs a guideline for protein folding. The few in the middle are the ones the scout will struggle with too.

Your turn: label the hard ones¶

Here are six papers where the right answer depends on the guideline. Before you look at the reference labels, decide each one yourself.

hard_ones = ["P03", "P09", "P11", "P15", "P17", "P23"]
by_id = {p["id"]: p for p in papers}

for pid in hard_ones:
    p = by_id[pid]
    print(f"{pid}: {p['title']}")
    print(f"   Abstract:  {p['abstract']}")
    print(f"   Full text: {p['full_text']}\n")

my_labels = {
    "P03": None,   # True or False
    "P09": None,
    "P11": None,
    "P15": None,
    "P17": None,
    "P23": None,
}

why = {
    "P03": "Retrieval practice is about human memory, a different sense of 'retrieval'.",
    "P09": "The full text shows it retrieves Wikipedia passages before answering.",
    "P11": "Product search ranks items; no language model writes from the results.",
    "P15": "It checks hallucinations without any documents.",
    "P17": "The agent writes its answer only from the web pages it opened.",
    "P23": "The code model writes from files retrieved and placed in its prompt.",
}

agree = 0
for pid, mine in my_labels.items():
    ref = by_id[pid]["label"]
    if mine is None:
        print(f"{pid}: not labeled yet (reference: {ref})")
        continue
    agree += (mine == ref)
    verdict = "agree" if mine == ref else "DISAGREE"
    print(f"{pid}: you {mine}, reference {ref} -> {verdict}. {why[pid]}")

labeled = sum(v is not None for v in my_labels.values())
print(f"\nAgreement: {agree} of {labeled} labeled")

P09 and P17 look irrelevant from the abstract alone; the full text settles both. If I needed the full text to label a paper, the scout will probably need it too. Keep an eye on those two.

One more check. Some gaps in a dataset only show up when you count:

for day in ["Mon", "Tue", "Wed"]:
    day_papers = [p for p in papers if p["day"] == day]
    relevant = sum(p["label"] for p in day_papers)
    print(f"{day}: {len(day_papers)} papers, {relevant} relevant")

no_relevant_days = [d for d in ["Mon", "Tue", "Wed"]
                    if not any(p["label"] for p in papers if p["day"] == d)]
print("\nDays with no relevant paper:", no_relevant_days or "none")

Every day has at least three relevant papers, so the dataset never tests a quiet day where the right answer is “skip everything”. I’m leaving that gap in on purpose. Every first dataset has gaps like this, and I’d rather you see one than pretend this one is complete.

chapter_3_actions = ["read_abstract", "read_full_text", "recommend", "nothing_today", "publish_summary"]

for action in chapter_3_actions:
    ok = action in scope_v1["allowed_actions"]
    print(f"{action:>16}: {'allowed' if ok else 'NOT ALLOWED in v1'}")

Three of the five don’t fit. recommend and nothing_today make one decision per day, while v1 needs a verdict per paper. And publish_summary is exactly what the scope says v1 must never do.

So far the scope is a document and the dataset is a list. Next chapter, v1 gets built to match the scope, with the “never” list enforced in code rather than requested in the prompt.

Quick check¶

Before moving on, answer for yourself (no peeking ahead):

Why does v1 start at the “suggest” setting instead of “draft” or “act”?

Chapter 3’s scout picked one paper per day. Why does v1 give every paper its own verdict?

Where did the guideline’s “NOT RELEVANT: retrieval with no language model writing from the results” rule come from, and why write it down at all?

You needed the full text to label P09 and P17 correctly. What does that suggest about how the scout should handle them?

The dataset has no day without relevant papers. Name one scout behavior this gap leaves untested, and one way to fix the gap.

You can now: explain the build-then-calibrate lifecycle, write a narrow scope with an explicit “never” list, and label papers against a written guideline.

But one problem remains: the scope and the labels describe a scout that doesn’t exist yet, and the one we have from Chapter 3 can still publish. The next chapter builds v1 exactly to scope, with the forbidden actions removed in code, and records what it does on every run.

Next → Chapter 5: Set Up v1