Chapter 2: Letting a Model Decide - Velmurugan Narayanasamy

Show the two day-4 abstracts from Chapter 1 to any researcher and they’ll sort them in five seconds. One searches over network designs; the other makes a model answer from fetched documents. Nobody needs a keyword list for that. They just read what each paper does.

Reading a short text and judging what it’s about is exactly what a language model is good at. So this chapter takes the decide function from Chapter 1 and hands it to a model.

There’s an optional live demo near the end that uses Google’s Gemini API, with the same free setup as Grounded AI Chapter 4: no credit card, just a key from aistudio.google.com/apikey. Everything before that runs right here in the page.

Chapter 2 of 12 · Foundations

Where we left off: Our scout runs a think, act, observe loop, but the “think” step is a list of hand-written keyword rules that match words, not topics.

What we fix now: Hand the “think” step to a language model: describe what you care about in a sentence, show it the papers, and get back a decision your code can act on.

By the end, you can:

explain which part of the agent a model replaces, and which parts stay ordinary code
build a prompt that asks for a structured decision, and read the reply safely before acting on it
identify why a model that can only see what’s in its prompt still gets stuck

The fix: the model proposes, your code decides what happens¶

The model’s job here is narrow. It reads text and writes text back: a proposal for what to do. It doesn’t run searches or write files. Your code reads that proposal, checks it, and carries it out, or doesn’t. I’ll keep coming back to this split for the rest of the course, because almost every safety decision later on depends on it.

The change from Chapter 1 is small in code:

Instead of a keyword list, describe the interest in a sentence, the way you’d explain it to a colleague.
Put that sentence and today’s papers into a prompt.
Ask for a reply in a fixed format your code can read: which paper to pick (or none), and one sentence saying why.
Your code reads the reply and acts on it: summarize, or stop.

Why a fixed format? Because the next step is code, and code can’t act on “I’d probably go with the second one, it looks more relevant”. It can act on {"pick": 1, "reason": "..."}. I ask for JSON by hand here so you can see where it goes wrong; Chapter 12 uses the model’s built-in structured output instead.

import json
import re

def summarize(paper):
    return paper["abstract"].split(". ")[0] + "."

# The same made-up papers as Chapter 1.
day_2 = [
    {"title": "Sparse Mixture-of-Experts at Scale",
     "abstract": "We route each token to 2 of 64 experts and train a 40B parameter model. Training cost falls by 30 percent."},
    {"title": "Protein Folding with Diffusion Models",
     "abstract": "We generate protein structures with a diffusion model. Predicted structures match lab results on 71 percent of targets."},
    {"title": "RAG Without Chunking",
     "abstract": "We show that RAG systems can skip chunking by indexing whole sections with a structure-aware encoder. Answer accuracy matches chunked baselines on two QA datasets."},
]
day_3 = [
    {"title": "Weather Forecasting with Graph Networks",
     "abstract": "We forecast 10 day weather on a global graph. Error falls by 12 percent against the previous best model."},
]
day_4 = [
    {"title": "Neural Architecture Search for Tiny Devices",
     "abstract": "We search over 10,000 network designs to find models that fit on a microcontroller. The best design runs in 40 milliseconds."},
    {"title": "Grounding Answers in External Documents",
     "abstract": "We make a language model answer only from passages fetched from a document store. Unsupported claims fall by 60 percent."},
]

interest = ("papers about retrieval-augmented generation (RAG), or about "
            "making language models answer from documents they are given")

def build_prompt(papers, interest):
    listing = "\n\n".join(
        f"[{i}] {p['title']}\n{p['abstract']}" for i, p in enumerate(papers)
    )
    return f"""You help a researcher decide which new paper to read today.
Their interest: {interest}

Today's papers:
{listing}

Reply with JSON only, in exactly this shape:
{{"pick": <paper number, or null if none match the interest>, "reason": "<one sentence>"}}"""

print(build_prompt(day_4, interest))

That’s everything the model will ever know about the task: one sentence of interest, the papers, and the reply format. Remember that; it’s the problem at the end of this chapter.

Plugging a model into the scout¶

A real model needs a network connection and an API key, and this page has neither. So the next cell uses a stand-in: mock_model returns a prewritten reply for each day. I wrote those replies by hand to look like typical model answers; they are not recorded from a real run. I’d rather show you a believable reply that behaves the same every time than a real one that changes on every page load. The Colab section at the end swaps in a real Gemini model with one line.

That one line matters: to the scout, a model is just a function that takes a prompt and returns text.

# Hand-written stand-in replies (NOT recorded from a real model).
prewritten_replies = {
    "Sparse Mixture-of-Experts at Scale":
        '{"pick": 2, "reason": "Paper 2 is about RAG systems, which matches the interest directly."}',
    "Weather Forecasting with Graph Networks":
        '{"pick": null, "reason": "The only paper is about weather forecasting, which has nothing to do with RAG or answering from documents."}',
    "Neural Architecture Search for Tiny Devices":
        '{"pick": 1, "reason": "Paper 1 makes a model answer from fetched passages, which is grounding; paper 0 searches over network designs, not documents."}',
}

def mock_model(prompt):
    for first_title, reply in prewritten_replies.items():
        if first_title in prompt:
            return reply
    return '{"pick": null, "reason": "No prewritten reply for these papers."}'

ask_model = mock_model   # the Colab section swaps this for a real model

def decide(papers):
    reply = ask_model(build_prompt(papers, interest))   # think
    return json.loads(reply)

def run_scout(papers):
    decision = decide(papers)
    print("Model's reason:", decision["reason"])
    if decision["pick"] is None:                         # act
        print("Stop -> Nothing relevant today.")
    else:
        paper = papers[decision["pick"]]
        print(f"Summarize '{paper['title']}'")
        print("  Summary:", summarize(paper))

run_scout(day_4)

Day 4 goes the right way, and the reason explains why: one paper fetches passages to ground answers, the other searches over network designs. No keyword rule could have made that call.

Try run_scout(day_2) and run_scout(day_3) too. On day 3 the model says none of them fit, and the scout stops, like the loop in Chapter 1, except nobody wrote a rule for it.

Reading the model’s reply safely¶

There’s a quiet assumption in decide: that the model replies with clean JSON and a paper number that actually exists. Real models usually do. Not always.

chatty_reply = 'Sure! Here is my decision:\n```json\n{"pick": 1, "reason": "Paper 1 is about grounding answers in documents."}\n```'
wrong_number_reply = '{"pick": 7, "reason": "Paper 7 is the most relevant."}'

for reply in [chatty_reply, wrong_number_reply]:
    try:
        decision = json.loads(reply)
        print("Parsed:", decision)
        print("  Paper:", day_4[decision["pick"]]["title"])
    except Exception as error:
        print("Crashed:", type(error).__name__, "-", error)

Both crash, for different reasons. The chatty reply isn’t valid JSON as a whole, even though a perfectly good decision is sitting inside it. The second parses fine, but paper 7 doesn’t exist, so it crashes one step later, when the code tries to use it.

I treat every model reply as data to check, never as a command to follow. Here’s a parse_reply that pulls the JSON out of the text, checks the paper number, and falls back to stopping when something doesn’t make sense:

def parse_reply(reply, papers):
    match = re.search(r"\{.*\}", reply, re.DOTALL)        # find the JSON inside the text
    if not match:
        return {"pick": None, "reason": "Could not find a decision in the model's reply."}
    try:
        decision = json.loads(match.group())
    except json.JSONDecodeError:
        return {"pick": None, "reason": "The model's reply was not valid JSON."}
    pick = decision.get("pick")
    if pick is not None and not (isinstance(pick, int) and 0 <= pick < len(papers)):
        return {"pick": None, "reason": f"The model picked paper {pick}, which doesn't exist."}
    return decision

for reply in [chatty_reply, wrong_number_reply]:
    print(parse_reply(reply, day_4))

def decide(papers):
    reply = ask_model(build_prompt(papers, interest))    # think
    return parse_reply(reply, papers)                     # check before acting

run_scout(day_4)

The chatty reply now works, and the impossible pick turns into a safe stop with a reason attached. decide checks every reply before the scout acts on it. This check will keep growing through the course; Chapter 12’s version also validates against a schema.

day_5 = [
    {"title": "Better Answers with Less Compute",
     "abstract": "We introduce a simple method that improves answer quality on two question answering benchmarks while using less compute. Code is available online."},
    {"title": "Faster Training for Speech Models",
     "abstract": "We cut speech model training time in half with a new batching scheme. Word error rate stays the same."},
]

prewritten_replies["Better Answers with Less Compute"] = (
    '{"pick": null, "reason": "Paper 0 might use retrieval, but its abstract never says '
    'how answers are improved, so I cannot tell from what I was given."}'
)

run_scout(day_5)

The stand-in says it can’t tell, and a careful real model often answers the same way. That’s a better answer than a guess. But a person would just open the paper and skim the method section. The answer is right there.

The scout can’t do that. Everything it knows was pasted into the prompt before it started, and when that isn’t enough, its only options are “pick”, “none”, or a guess. It gets worse at scale: a real arXiv day has over a hundred new papers, and pasting every full text into one prompt isn’t an option.

Bonus (Colab only): a real model decides¶

Everything above runs anywhere, because mock_model is just a dictionary lookup. To see a real model make these decisions, open this notebook in Colab and run the cell below:

Get a free key at aistudio.google.com/apikey (no credit card needed), then paste it into API_KEY below. Treat it like a password: fine to paste into your own private Colab copy while experimenting, never commit it to a public repository or notebook.

# Colab only: needs a live network call and an API key.
!pip install -q -U google-genai

from google import genai

API_KEY = "YOUR_API_KEY_HERE"
client = genai.Client(api_key=API_KEY)
MODEL = "gemini-3.5-flash"

def gemini_model(prompt):
    response = client.models.generate_content(model=MODEL, contents=prompt)
    return response.text

ask_model = gemini_model   # the one-line swap

for name, papers in [("day 2", day_2), ("day 3", day_3), ("day 4", day_4), ("day 5", day_5)]:
    print(f"=== {name} ===")
    run_scout(papers)
    print()

Compare the real reasons with the stand-in ones. The wording will differ, and on day 5 a real model might pick paper 0 on a hopeful guess instead of saying it can’t tell. Run it a few times and the answers can change. That variation is the reason the second half of this course exists.

Quick check¶

Before moving on, answer for yourself (no peeking ahead):

Of the three moves, think, act, observe, which one did the model take over? What stayed ordinary Python, and why does that split matter?

Chapter 1 used a keyword list. This chapter describes the interest in a sentence. Why does that fix day 4, where the keyword rules failed?

Why does the prompt ask for JSON in a fixed shape instead of letting the model answer freely?

parse_reply turned “pick paper 7” into a safe stop. Why should code check a model’s decision instead of just trusting it?

On day 5 the model said it couldn’t tell. What would a person do next, and what would the scout need in order to do the same thing?

You can now: explain which part of an agent a model replaces, build a prompt that returns a structured decision, and check that decision before your code acts on it.

But one problem remains: the model can only judge what was pasted into its prompt. When an abstract doesn’t say enough, it can’t go and read more, and on a real day there are far too many papers to paste in at once. The next chapter gives the model actions it can ask for, like searching papers and reading one in full, and then asks an uncomfortable question: once it can act, how much should it be allowed to do without you?

Next → Chapter 3: Tools and the Autonomy Dial