The whole rulebook, at the speed of a question

The corpus problem

Solvency II’s Delegated Regulation runs to hundreds of pages of dense, cross-referential text, and an honest answer to a rulebook question usually involves three articles and a definition from somewhere else entirely. Frontier models read fast, but left to themselves they recall confidently and cite approximately. The useful pattern is retrieval: find the passages first, answer only from them, and quote as you go. The corpus here is entirely public — the point is the pattern, which transfers to any long technical text.

The pattern

Five decisions carry almost all of the quality:

1. Chunk by structure, not by tokens. Split on article boundaries, and keep each article’s number and hierarchy attached to its text. Naive fixed-length chunks cut definitions away from their uses.
2. Index with context. Embed each article with its position — part, chapter, section — so “mass lapse under the standard formula” lands near the right mass lapse.
3. Retrieve with a floor. Return the top few passages by similarity, and below a similarity floor return nothing; abstention must be cheap.
4. Answer from quotations. The drafting model is shown the retrieved passages and instructed to cite article numbers for every claim and to say plainly when the passages do not answer the question.
5. Stamp every chunk with its source version. Rules change; an answer from last year’s text must know it.

A working skeleton is genuinely small — the helpers are any embedding and similarity function; the decisions above are the substance:

import re
from pathlib import Path

ARTICLE = re.compile(r"^(Article \d+[a-z]?\b.*)$", re.MULTILINE)

def chunks(path):
    """Yield one chunk per article, keeping the article number with its text."""
    parts = ARTICLE.split(Path(path).read_text(encoding="utf-8"))
    for i in range(1, len(parts) - 1, 2):
        yield {"ref": parts[i].strip(), "text": parts[i] + parts[i + 1].strip()}

index = [(c["ref"], embed(c["text"]), c["text"]) for c in chunks("delegated_regulation.txt")]

def retrieve(question, k=6, floor=0.35):
    """Top-k passages by cosine similarity; nothing if below the floor."""
    scored = sorted(index, key=lambda row: -cosine(embed(question), row[1]))
    return [(r, t) for r, v, t in scored[:k] if cosine(embed(question), v) >= floor]

PROMPT = """Answer only from the passages quoted below.
Cite the article number for every claim; quote before you conclude.
If the passages do not answer the question, say so plainly.

Passages:
{passages}

Question: {question}"""

A day, not a month — another small datum for what the tooling now costs.

Failure modes that matter

• Chunk-boundary amnesia. The term is defined in one article and used in another forty pages later. Structural chunking plus attached context mitigates it; nothing abolishes it, so cross-references deserve their own pass.
• Citation theatre. An answer flanked by a citation whose passage does not support it. The quote-then-conclude discipline catches most of it, because a claim that cannot point at quoted words is visibly naked.
• Version drift. A correct answer about a superseded text is a wrong answer with better manners. Version stamps on every chunk, and the stamp belongs in the reply.
• Confident absence. “Nothing in the rules requires X” needs far stronger retrieval evidence than any presence claim. Treat absence answers as a separate, harder mode and label them as such.

The evaluation habit

Keep a golden set: questions whose correct answers a human has verified once, run against the pipeline whenever the chunking, the index or the drafting model changes. Track hit-rate and citation precision — the share of cited passages that actually support their claims. It is the same regression habit as the engines on this site, in different clothing: a canonical run, pinned, so that any change which moves a number must justify itself.

Where else it serves

The same pattern turns any long corpus into something an actuary can interrogate: methodology libraries, validation reports, internal policy documents, the accumulated minutes of an assumptions committee. Make the text findable, make the answers quotable, and make abstention cheap. Trust then scales with the corpus, which is the one thing about these tools that was never going to arrive on its own.