Twelve chapters, one running example: a paper scout that reads new AI papers and suggests which ones are worth my time. I built it because picking a paper for Vector Reads every week starts with skimming a hundred abstracts, and I wanted to see how far a small agent could take that over.
Every chapter starts from something that breaks, looks at why, adds one fix, and runs into the next problem. The first three chapters build the agent. The rest follow the CC/CD lifecycle (Continuous Development and Continuous Calibration, from Aishwarya Reganti and Kiriti Badam’s article Why your AI product needs a different development lifecycle): build the narrowest useful version, measure it, watch it on real use, find the patterns in its mistakes, fix them, and only then let it do more on its own. The last chapter deploys it on live arXiv papers with a real model.
Chapters 1 to 11 run in your browser with hand-written stand-in model replies, so every reader hits the same failures. Chapter 12 needs Colab and a free Gemini API key.
Foundations · Chapters 1-3¶
The loop, the model, and the tools: the minimum an agent needs before it can do anything useful, and the first question about how much it should do without you.
A script with fixed steps summarizes the wrong paper when its results surprise it, because it never looks before taking the next step.
Hand-written rules match words, not topics, and every new kind of paper needs another rule.
A model that can only judge what’s in its prompt can’t go and read more, and once it can act, one good run says nothing about the next.
Continuous Development · Chapters 4-6¶
Building on purpose: a narrow scope, labeled examples, a first version that stays inside its limits, and a real measurement before shipping.
Tweaking the prompt after every bad run means steering by the last example you happened to look at.
A written scope and a list of labels don’t do anything until a scout exists that follows them.
“The suggestions look fine” is an impression, and it can’t see a single missed paper.
Continuous Calibration · Chapters 7-10¶
What happens after shipping: measuring real use, reading mistakes until they turn into patterns, fixing without breaking, and earning the next level of autonomy.
An eval only covers the papers you chose, and real use brings papers, and problems, you never thought of.
Two falling numbers say that something is wrong, but not what.
Every fix changes the system the other mistakes live in, and some fixes break what used to work.
A scout can beat the old version on every number you track and still get worse at something none of them measure.
Capstone · Chapters 11-12¶
The whole loop one more time, one level up, and then the real thing: the scout deployed on live papers, measured on real use.