Chapter 5: Set Up v1 - Velmurugan Narayanasamy

"never": ["publish anything", "email anyone", "change the reading list"]

That line from the v1 scope is the one I care about most, and the scout from Chapter 3 breaks it: it still has a publish_summary action in its toolbox, and it picks one paper per day instead of judging every paper.

This chapter builds the scout the scope describes, and nothing more. Most of it is plain. The part worth slowing down for is how v1 is kept inside its scope. I could write “never publish anything” in the prompt, and a good model would usually listen. For a rule whose whole job is to limit damage, usually isn’t enough.

Chapter 5 of 12 · Continuous Development

Where we left off: We have a written scope and 24 labeled papers, but no scout that matches the scope.

What we fix now: Build v1 exactly to scope, with its limits enforced by code, and record a trace of everything it does.

By the end, you can:

build a scout whose allowed actions come straight from the scope, so forbidden actions simply don’t exist
record a trace for every paper, showing each step the scout took and why
explain why looking at the scout’s suggestions can’t tell you how good it is

The fix: limits in code, not in the prompt¶

There are two ways to stop v1 from publishing:

Ask nicely. Put “never publish anything” in the prompt.
Remove the ability. Don’t give v1 a publish tool at all, and refuse any action that isn’t on the scope’s list.

The first depends on the model following an instruction every single time. The second doesn’t depend on the model at all: if v1 proposes publishing, there’s nothing to run, so the code refuses. I still tell the model which actions exist, because that helps it choose well. The guarantee comes from the code.

It’s the same split as Chapter 2, one level up: the model proposes, the code decides what actually happens.

papers = [
    {"id": "P01",
     "day": "Mon",
     "title": "RAG Without Chunking",
     "abstract": "We show that RAG systems can skip chunking by indexing whole sections with a structure-aware encoder. Answer accuracy matches chunked baselines on two QA datasets.",
     "full_text": "Each document section is embedded as one unit, retrieved by similarity, and passed to the language model as context for answering."},
    {"id": "P02",
     "day": "Mon",
     "title": "Sparse Mixture-of-Experts at Scale",
     "abstract": "We route each token to 2 of 64 experts and train a 40B parameter model. Training cost falls by 30 percent.",
     "full_text": "A learned router sends each token to two expert networks, so only a small part of the model runs per token."},
    {"id": "P03",
     "day": "Mon",
     "title": "Retrieval Practice Boosts Exam Scores in Engineering Courses",
     "abstract": "Students who practiced retrieving facts from memory scored 14 percent higher on final exams than students who reread their notes. We studied 300 students across four courses.",
     "full_text": "Retrieval practice means recalling information from memory, for example with flashcards, instead of rereading. No computer systems are involved."},
    {"id": "P04",
     "day": "Mon",
     "title": "Grounding Answers in External Documents",
     "abstract": "We make a language model answer only from passages fetched from a document store. Unsupported claims fall by 60 percent.",
     "full_text": "Passages are fetched with a dense retriever and the model is trained to refuse when they do not contain the answer."},
    {"id": "P05",
     "day": "Mon",
     "title": "Protein Folding with Diffusion Models",
     "abstract": "We generate protein structures with a diffusion model. Predicted structures match lab results on 71 percent of targets.",
     "full_text": "A diffusion model denoises random 3D coordinates into a protein backbone."},
    {"id": "P06",
     "day": "Mon",
     "title": "A Smaller Vision Transformer",
     "abstract": "We prune a vision transformer to a third of its size. Accuracy drops by less than one point.",
     "full_text": "Attention heads with low importance scores are removed and the model is briefly retrained."},
    {"id": "P07",
     "day": "Mon",
     "title": "Citation Accuracy in LLM Answers",
     "abstract": "We measure how often cited passages actually support the sentences that cite them. Only 64 percent of citations fully support their claims.",
     "full_text": "Answers are generated from retrieved passages with inline citations, and annotators check each citation against its passage."},
    {"id": "P08",
     "day": "Mon",
     "title": "Faster Training for Speech Models",
     "abstract": "We cut speech model training time in half with a new batching scheme. Word error rate stays the same.",
     "full_text": "Audio clips of similar length are grouped into the same batch so less padding is wasted."},
    {"id": "P09",
     "day": "Tue",
     "title": "Better Answers with Less Compute",
     "abstract": "We introduce a simple method that improves answer quality on two question answering benchmarks while using less compute. Code is available online.",
     "full_text": "Before answering, our method retrieves the five most similar passages from a Wikipedia index and adds them to the prompt. Because the answer comes from the passages, a much smaller model is enough."},
    {"id": "P10",
     "day": "Tue",
     "title": "Neural Architecture Search for Tiny Devices",
     "abstract": "We search over 10,000 network designs to find models that fit on a microcontroller. The best design runs in 40 milliseconds.",
     "full_text": "An evolutionary search mutates network designs and keeps the ones that meet a latency budget."},
    {"id": "P11",
     "day": "Tue",
     "title": "Dense Retrieval for Product Search",
     "abstract": "We train a dense retriever that ranks products for shopping queries. Click-through rate rises by 9 percent in an online test.",
     "full_text": "Queries and product descriptions are embedded in one vector space, and the top products are shown to the shopper. No language model generates text."},
    {"id": "P12",
     "day": "Tue",
     "title": "Long-Context Models vs. Retrieval: When Is RAG Still Needed?",
     "abstract": "We compare feeding whole documents into long-context models with retrieving a few passages. Retrieval wins on cost and matches accuracy for most questions.",
     "full_text": "Both setups answer the same questions over the same documents, and we measure accuracy, latency and cost."},
    {"id": "P13",
     "day": "Tue",
     "title": "Teaching Robots to Fold Laundry",
     "abstract": "We train a robot arm to fold shirts from 200 demonstrations. It succeeds on 8 of 10 unseen garments.",
     "full_text": "A policy network maps camera images to arm motions and is trained by imitation."},
    {"id": "P14",
     "day": "Tue",
     "title": "Weather Forecasting with Graph Networks",
     "abstract": "We forecast 10 day weather on a global graph. Error falls by 12 percent against the previous best model.",
     "full_text": "The globe is split into a mesh of nodes and a graph network predicts the next state of each node."},
    {"id": "P15",
     "day": "Tue",
     "title": "Hallucination Detection via Self-Consistency",
     "abstract": "We detect hallucinated answers by sampling a model several times and checking whether the samples agree. Detection accuracy reaches 81 percent.",
     "full_text": "No documents are used: the model is compared only against its own repeated answers."},
    {"id": "P16",
     "day": "Tue",
     "title": "Reranking Retrieved Passages with Small Models",
     "abstract": "We show that a small cross-encoder can rerank retrieved passages before they reach the generator. RAG answer accuracy rises by 5 points.",
     "full_text": "The retriever returns 50 passages, the cross-encoder keeps the best 5, and the language model answers from those."},
    {"id": "P17",
     "day": "Wed",
     "title": "Web-Browsing Agents that Answer from Live Pages",
     "abstract": "We build an agent that browses the web and answers questions. It beats a closed-book model by 18 points.",
     "full_text": "The agent searches the web, opens the top pages, and writes its answer only from the text of those pages, citing each one."},
    {"id": "P18",
     "day": "Wed",
     "title": "Quantizing LLMs to 3 Bits",
     "abstract": "We compress a language model's weights to 3 bits with little loss in quality. Memory use falls by 80 percent.",
     "full_text": "Weights are grouped and each group gets its own scale factor before rounding."},
    {"id": "P19",
     "day": "Wed",
     "title": "Evaluating RAG Pipelines Without Human Labels",
     "abstract": "We score RAG answers for faithfulness and relevance using a judge model instead of human labels. Scores agree with humans 87 percent of the time.",
     "full_text": "A judge model reads the question, the retrieved passages and the answer, and rates whether the answer is supported."},
    {"id": "P20",
     "day": "Wed",
     "title": "Graph Neural Networks for Traffic",
     "abstract": "We predict traffic speed on city roads with a graph neural network. Error falls by 8 percent.",
     "full_text": "Road sensors are nodes and roads are edges in the graph."},
    {"id": "P21",
     "day": "Wed",
     "title": "Chunk Size Matters: A Study of Document Splitting",
     "abstract": "We test chunk sizes from 128 to 2048 tokens across six RAG datasets. Mid-sized chunks work best for most questions.",
     "full_text": "Documents are split at each size, indexed, and used for retrieval-augmented answering."},
    {"id": "P22",
     "day": "Wed",
     "title": "Music Generation with Transformers",
     "abstract": "We generate piano music with a transformer trained on 10,000 hours of recordings. Listeners preferred it 60 percent of the time.",
     "full_text": "Audio is converted to tokens and the transformer predicts the next token."},
    {"id": "P23",
     "day": "Wed",
     "title": "Code Completion with Repository Context",
     "abstract": "We retrieve related files from the same repository and add them to the prompt of a code model. Completion accuracy rises by 11 points.",
     "full_text": "Files are ranked by similarity to the current file and the top three are placed in the prompt before completing code."},
    {"id": "P24",
     "day": "Wed",
     "title": "Speeding Up Diffusion Sampling",
     "abstract": "We reduce diffusion sampling from 50 steps to 8 steps. Image quality stays close to the original.",
     "full_text": "A student model is trained to match several teacher steps at once."},
]

scope_v1 = {
    "job": "For each paper, decide suggest or skip, with a one-sentence reason.",
    "autonomy": "suggest",
    "allowed_actions": ["read_abstract", "read_full_text", "suggest", "skip"],
    "never": ["publish anything", "email anyone", "change the reading list"],
}
interest = ("papers about retrieval-augmented generation (RAG), or about "
            "making language models answer from documents they are given")
by_id = {p["id"]: p for p in papers}
print(f"Loaded {len(papers)} papers. v1 may: {scope_v1['allowed_actions']}")

Now v1 itself, in three parts:

Actions come from the scope. TOOLS only has functions for actions the scope allows. publish_summary isn’t defined anywhere, and execute refuses anything not in scope_v1["allowed_actions"].
One paper at a time. judge_paper runs the loop for a single paper until the model proposes suggest or skip, with a confidence between 0 and 1.
Everything is recorded. Each step goes into a trace, and each finished paper goes into log. I log more than I need right now: adding a field later doesn’t help with the runs that already happened.

import json
import re

TOOLS = {
    "read_abstract": lambda p: p["abstract"],
    "read_full_text": lambda p: p["full_text"],
}
VERDICTS = {"suggest", "skip"}

def build_prompt(paper, transcript):
    steps = "\n".join(transcript) if transcript else "(nothing yet)"
    return f"""You help a researcher decide whether to read a new paper.
Their interest: {interest}

Paper id: {paper['id']}
Title: {paper['title']}

Actions: read_abstract, read_full_text (when the abstract is not enough),
suggest (it matches the interest), skip (it does not).

What you have done so far:
{steps}

Reply with JSON only:
{{"action": "<action>", "confidence": <0 to 1, only for suggest or skip>, "reason": "<one sentence>"}}"""

def execute(action, paper):
    if action not in scope_v1["allowed_actions"]:
        return f"Refused: '{action}' is not allowed in v1."
    return TOOLS[action](paper)

def parse(reply):
    match = re.search(r"\{.*\}", reply, re.DOTALL)
    try:
        return json.loads(match.group()) if match else {"action": "invalid"}
    except json.JSONDecodeError:
        return {"action": "invalid"}

log = []

def judge_paper(paper, max_steps=5):
    transcript, trace = [], []
    for step in range(1, max_steps + 1):
        d = parse(ask_model(build_prompt(paper, transcript)))            # think
        action = d.get("action")
        if action in VERDICTS:
            record = {"id": paper["id"], "day": paper["day"], "verdict": action,
                      "confidence": d.get("confidence"), "reason": d.get("reason"), "trace": trace}
            log.append(record)
            return record
        result = execute(action, paper)                                    # act (or refuse)
        trace.append(f"{action} <- {d.get('reason')}")
        if result.startswith("Refused"):
            trace.append(result)
            print(f"   [{paper['id']}] {result}")
        transcript.append(f"Step {step}: {action}\nResult: {result}")     # observe
    record = {"id": paper["id"], "day": paper["day"], "verdict": "skip",
              "confidence": 0.0, "reason": "Ran out of steps.", "trace": trace}
    log.append(record)
    return record

def run_v1(day):
    suggestions = []
    for paper in [p for p in papers if p["day"] == day]:
        record = judge_paper(paper)
        if record["verdict"] == "suggest":
            suggestions.append(record)
    print(f"v1 suggestions for {day}:")
    for r in sorted(suggestions, key=lambda r: -r["confidence"]):
        print(f"  {r['id']}  ({r['confidence']:.2f})  {by_id[r['id']]['title']}")
        print(f"         {r['reason']}")
    print("You decide what to read. Nothing else happens.")

print("v1 is ready.")

v1 still needs a model, so the page uses a hand-written stand-in again. This one reads from a table with one entry per paper: its confidence, its reason, and whether it opens the full text. I wrote the table for this course and left realistic mistakes in it on purpose, so Chapter 6 has something real to measure. It is not recorded from a real model; the Colab bonus runs a real one.

stand_in_verdicts = {
    "P01": {"confidence": 0.91, "reason": "The paper is about RAG systems and how they index documents."},
    "P02": {"confidence": 0.05, "reason": "The paper is about model architecture and training cost, not documents."},
    "P03": {"confidence": 0.62, "reason": "The paper is about retrieval, which matches the interest."},
    "P04": {"confidence": 0.88, "reason": "The model answers only from fetched passages, which is grounding in documents."},
    "P05": {"confidence": 0.03, "reason": "The paper is about proteins, unrelated to documents or answering."},
    "P06": {"confidence": 0.04, "reason": "The paper is about image models, not language models answering from documents."},
    "P07": {"confidence": 0.81, "reason": "The paper checks whether answers are supported by the passages they cite."},
    "P08": {"confidence": 0.06, "reason": "The paper is about speech model training, not documents."},
    "P09": {"confidence": 0.4, "reason": "The abstract does not mention documents or retrieval, so it is probably not a match."},
    "P10": {"confidence": 0.08, "reason": "The search here is over network designs, not documents."},
    "P11": {"confidence": 0.58, "reason": "The paper improves dense retrieval, a core part of RAG."},
    "P12": {"confidence": 0.86, "reason": "The paper compares RAG with long-context models for answering from documents."},
    "P13": {"confidence": 0.02, "reason": "The paper is about robotics."},
    "P14": {"confidence": 0.03, "reason": "The paper is about weather forecasting."},
    "P15": {"confidence": 0.35, "reason": "The paper is about hallucinations, but it does not use documents."},
    "P16": {"confidence": 0.79, "reason": "Reranking passages before generation is a part of a RAG system."},
    "P17": {"confidence": 0.45, "reason": "The abstract is about a browsing agent, which is not clearly about documents."},
    "P18": {"confidence": 0.07, "reason": "The paper is about compressing models, not documents."},
    "P19": {"confidence": 0.84, "reason": "The paper evaluates RAG pipelines."},
    "P20": {"confidence": 0.04, "reason": "The paper is about traffic prediction."},
    "P21": {"confidence": 0.77, "reason": "Chunking documents is a part of RAG systems."},
    "P22": {"confidence": 0.03, "reason": "The paper is about music generation."},
    "P23": {"confidence": 0.66, "reason": "The model writes code using retrieved files from the repository, which is answering from given documents."},
    "P24": {"confidence": 0.05, "reason": "The paper is about image generation speed."},
}

reads_full_text = {"P16", "P23"}        # the only papers where the stand-in opens the full text
tries_to_publish = {"P19"}              # the stand-in tries a forbidden action here

def stand_in_script(pid):
    v = stand_in_verdicts[pid]
    steps = ['{"action": "read_abstract", "reason": "Start with the abstract."}']
    if pid in reads_full_text:
        steps.append('{"action": "read_full_text", "reason": "I want to check the method before deciding."}')
    if pid in tries_to_publish:
        steps.append('{"action": "publish_summary", "reason": "This is clearly relevant, so I will share it right away."}')
    verdict = "suggest" if v["confidence"] >= 0.5 else "skip"
    steps.append(json.dumps({"action": verdict, "confidence": v["confidence"], "reason": v["reason"]}))
    return steps

def mock_model(prompt):
    pid = re.search(r"Paper id: (\w+)", prompt).group(1)
    script = stand_in_script(pid)
    return script[min(prompt.count("Result:"), len(script) - 1)]

ask_model = mock_model
run_v1("Mon")

That’s v1 on Monday: four suggestions with confidences and reasons, and a last line that sums up the “suggest” setting. The scout suggests; I decide.

If something in Monday’s list already looks off to you, hold on to it. First, the forbidden action getting refused.

run_v1("Wed")

On P19 the stand-in tried publish_summary. However sure it was, the action isn’t in the scope, so execute refused it, the refusal went into the trace, and the scout went on to a normal verdict.

Every decision leaves a trace like this. Here’s how to read one:

def show_trace(pid):
    r = next(r for r in reversed(log) if r["id"] == pid)
    print(f"{pid}: {by_id[pid]['title']}")
    for step in r["trace"]:
        print("   ", step)
    print(f"    -> {r['verdict']} ({r['confidence']:.2f}): {r['reason']}")

show_trace("P23")
print()
show_trace("P19")

Traces look like bookkeeping now. In Chapter 8 they’re the main tool for working out why the scout gets things wrong. Try show_trace("P03"): what did the scout look at before deciding?

run_v1("Tue")

You can judge the three papers on the list, and maybe product search (P11) already looks wrong. But the question was about papers v1 missed, and those aren’t on the list. A skipped paper is invisible in the output. Counting misses means going through every Tuesday paper, including the five it skipped, and checking each against the right answer. That’s what Chapter 4’s labels are for.

Bonus (Colab only): a real model as v1¶

Open this notebook in Colab to run v1 with a real model:

Get a free key at aistudio.google.com/apikey (no credit card needed), then paste it into API_KEY below. Treat it like a password: fine to paste into your own private Colab copy while experimenting, never commit it to a public repository or notebook.

# Colab only: needs a live network call and an API key.
!pip install -q -U google-genai

from google import genai

API_KEY = "YOUR_API_KEY_HERE"
client = genai.Client(api_key=API_KEY)
MODEL = "gemini-3.5-flash"

def gemini_model(prompt):
    return client.models.generate_content(model=MODEL, contents=prompt).text

ask_model = gemini_model
log = []
for day in ["Mon", "Tue", "Wed"]:
    run_v1(day)
    print()

A real model will behave differently from the stand-in: other confidences, maybe other mistakes, and it may open full texts more often. Keep its log. Chapter 6’s evaluation works the same way on a real model’s verdicts.

Quick check¶

Before moving on, answer for yourself (no peeking ahead):

v1 has no publish_summary tool, and execute refuses anything outside the scope. Why is that stronger than writing “never publish” in the prompt?

When the stand-in tried to publish on P19, what happened next, and why is it useful that the refusal went into the trace?

Why does judge_paper store every verdict in log, including skips?

Run show_trace("P03") and show_trace("P09"). What did the scout look at before deciding, and what does that have in common with what you needed when labeling them in Chapter 4?

Tuesday’s output showed three suggestions. Why can’t you tell from it how many relevant papers v1 missed?

You can now: build a scout whose limits come from its scope and are enforced in code, record a trace for every decision, and explain why reading its suggestions can’t tell you how good it is.

But one problem remains: “the suggestions look fine” is an impression, not a measurement, and it can’t see missed papers at all. The next chapter compares every one of v1’s verdicts against the labels, turns the result into numbers, and uses them to decide whether v1 is ready to ship.

Next → Chapter 6: Design Evals and Deploy