Working rules.
Fifteen years of building for regulated, sensitive-data environments leaves you with habits. These are mine, written down so any team, inside or outside those environments, can hold me to them. Like the case studies, this stays at the architectural altitude: principles and the controls that enforce them, no client specifics.
Evidence before confidence
An AI answer is a claim about the world, and claims carry obligations. Every system I ship attributes its statements to retrieved sources at claim level, and when the evidence doesn’t cover the question, it refuses with the gap named. Refusal is a success state: the system declining to spend your trust on a guess.
Cite-or-refuse gate in code, not in the promptprevents confident fabrication
A person owns the decision
Models draft, extract, and accelerate; they do not decide. Anything consequential passes a named human before it acts, and audit and replay exist so accountability has a trail rather than a shrug. If an outcome cannot be traced back to a person and the evidence they saw, the system isn’t finished.
Human sign-off, with audit and replayprevents accountability gaps
The least data, held the shortest time
I treat data protection as an engineering constraint, not compliance paperwork: collect what the purpose needs, use it for that purpose, retire it on schedule. GDPR-style principles (minimisation, purpose limitation, retention discipline) are design inputs from day one, because bolting them on later is how systems leak.
Minimisation and purpose-bound retentionprevents quiet data sprawl
Privacy by architecture
Sensitive data stays inside its boundary by default: local-first models where data cannot leave, air-gapped deployment where it must not, and never a third-party API call as the side effect of an innocent-looking feature. The safe path is the default path; anything else is a deliberate, recorded decision.
Local-first and air-gapped by defaultprevents silent exfiltration
Bias and failure honesty
Every evaluation scores both failure directions (fabricating under pressure and over-refusing what the evidence supports) because a system tuned against only one bias quietly maximises the other. Judging is blind, denominators are honest, and a release is compared to the one it replaces by failure class before anything ships.
Two-sided evaluation, judged blindprevents one-directional bias
How I evaluate production RAG systemsLines I hold
- No system that certifies its own output.
- No automation that hides its uncertainty from the person relying on it.
- No collection beyond the mandate, and no keeping data because it might be useful someday.
Further reading and source code: reference repo · writing.