Building an engine with an agent crew

The honest answer to the obvious question

Since the models on this site credit no team, the question arrives quickly: does one person really build this, and how? The honest answer is that most of the typing is done by systems that write code on instruction — frontier AI agents — and that everything determining whether the output is any good remains human. This piece describes the working arrangement: what I hand over, what I keep, and the discipline in between that separates an engine from a pile of plausible files.

It is the second piece on the economics of all this; the first, Build vs buy after the cost collapse, argues the why. This one is about the how.

Specification is the job

An agent given a vague instruction produces a fluent answer to a question of its own choosing. The correction is to treat specification as the primary work product. Every component of both engines began as documents: a charter stating scope and acceptance, a methodology document deriving the mathematics to be implemented, and — before any production code — a written account of the evidence that will prove the component works. These documents are not paperwork around the work; they are the steering mechanism. An agent pointed at a citation-grade methodology implements it within narrow error. An agent pointed at a vibe wanders, confidently, at speed.

The per-component cycle is fixed: charter, implement, review gate, end-to-end validation, verify gate, commit. End-to-end validation — the component run against real-shaped inputs with its evidence produced — is the unit of progress. Code written is not progress.

Review is the scarce resource

Writing is cheap now; review is not, and it is the input that gates quality. Two rules have mattered most.

The first: nothing is reviewed by its author. Every generation passes to a reviewer that has not written it — a fresh context shown only the artefacts and the specification, instructed to find fault — and then to me, with actuarial rather than stylistic questions. A writer asked to review its own work finds it excellent; this is as true of the machines as it is of us.

The second: the human gate is not ceremonial. I read the methodology derivation, the calibration path and the evidence output for every component, because the failure that matters is not the crash but the wrong answer that formats nicely. Catching it requires knowing what the number should smell like, which is the twenty years talking, not the tools.

The tripwire

Tests assert behaviour; fingerprints pin numbers. Every canonical run of both engines is locked to a bit-identical fingerprint — 155 pytest files in the ESG’s harness, and in the ALM engine a sealed, self-contained run bundle that any third party can re-run to the same bits. Any change that moves a locked number fails the suite and must justify itself in the commit. This is what makes agent-written code tolerable in a regulated context: enthusiasm is capped by evidence, and nothing ships because the explanation sounded right.

What the agents are good at, and what they are bad at

Good at: tirelessness across two hundred files at the same attention as the first; consistency of style; writing the tests first without being nagged; rewriting the same function four times without embarrassment.

Bad at: knowing when an answer is wrong but plausible; choosing the right convention when the market uses three (they pick one, confidently, and it may not be yours); restraint — left alone they gold-plate, generalise and add features nobody specified; and admitting uncertainty, which in an actuarial artefact is the one admission that matters.

The division of labour that works

I own the mathematics, the architecture, the acceptance criteria and every review gate. The agents draft, test, refactor and document under that direction, and their work lands only through the gates. The skills that transfer from twenty years of modelling turn out to be exactly the scarce ones: specification, verification, and knowing what a validator will ask next year.

What has changed, and what has not

Changed: who can build, and at what cost — one actuary, evenings and weekends, two engines. Not changed: what must be true of the thing built. An engine still has to be right, still has to prove it, and someone accountable still has to sign. The tools have moved the typing. They have not moved the responsibility, and the profession should distrust anyone who says otherwise.