Writing
Reviewing a model with an agent: a working method
model review · validation · agents
The bottleneck discipline
The cost of building a model has collapsed; the cost of trusting one has not, which leaves independent review as the bottleneck discipline of the next few years. It is also, awkwardly, the work the new tools assist least naturally: a review is an act of organised distrust, and the tools are optimised to be agreeable. Used with structure, though, they change the economics of review just as they changed the economics of building. This piece sets out the method I use: five steps, an explicit division of labour, and three traps that will catch anyone who skips the structure.
The method
1. Scope from documents, not from code. Read the methodology first and frame the review question: does the code implement these equations, under these conventions, and do both trace to the literature they cite? A model without a methodology document is not hard to review — it has failed already, at the gate.
2. Make the model reproduce itself. Before reading anything, run whatever harness exists. The password hygiene of modelling is reproducibility: if the model cannot reproduce its own published numbers on a fresh machine, that is finding one, and everything else is secondary.
3. Recompute independently. Re-derive the headline outputs by a deliberately second route — a spreadsheet oracle of the governing equations is often enough for a BEL or a calibration target — and compare against a stated tolerance. Agreement between two weak routes is weak evidence; agreement between two independent routes is where assurance begins.
4. Probe adversarially. Edge cases and degenerate inputs: empty portfolios, flat curves, zero volatility, the day before a coupon. Machines are superb at generating these tirelessly; the reviewer’s craft is choosing the few that hurt.
5. Trace the provenance of every assumption. Regulation, primary literature, or judgement — each tagged at source. An assumption without provenance is not wrong; it is unfinished, and the report should say so, one list at a time.
What to hand the machine
Reading without fatigue — two hundred files at the same attention as the first. Cross-checking documentation against code line by line, equation against function. Generating edge cases, challenge lists and first-pass drafts of findings. Machine hours are cheap; spend them there.
What to keep for yourself
Materiality. The acceptability judgement. The sign-off. The machine forms no risk assessment and owns no opinion; the reviewer’s name goes on the opinion, and the profession should distrust any review whose accountable human cannot reconstruct every step that led to it. A review that cannot be explained without the tool is not a review; it is a transcript.
The traps
- Plausibility bias. A fluent summary of wrong code reads exactly like a fluent summary of right code. Never let a summary substitute for the artefact; the machine points, the reviewer reads.
- Confirmation by construction. Do not ask the author to review its own work — and the author, increasingly, is a machine. The reviewing context must be cold: fresh, shown only the artefacts and the specification, instructed to find fault.
- Confidentiality hygiene. A client model, or data derived from one, does not go into an external tool. Run the review harness inside the client’s perimeter, on synthetic extracts. The constraint is not a limitation of the method; it is a rule of professional practice, and it disciplines the design for the better.
The public test
A review method should be willing to fail the reviewer’s own models. Every step above can be re-run against the engines published on this site: the methodology documents, the regression harnesses and the sealed evidence bundles exist precisely so that a sceptical actuary can scope, reproduce, recompute, probe and trace — and then say, with evidence, exactly where I am wrong.