Concept 8 min read
What a harness is — and why the model alone isn’t a product
Here is a demo anyone can build in an afternoon: take a scanned loan document, paste it into an AI chatbot, and ask “what’s the APR?” It will answer. Usually correctly. The demo is genuinely impressive, and it is genuinely an afternoon of work, because the hard part — the reading — is done by a model somebody else spent billions of dollars training.
So why isn’t that a product a bank can use?
Because everything the demo skipped is the product. The model can read a page, but it cannot promise to read it the same way tomorrow. It sometimes answers questions it should have refused. It occasionally reports a value the page never printed. It has no memory of what it did, no record anyone can audit, and no opinion about whether the document it just read should ever have left the building. A bank doesn’t need a brilliant reader; it needs a reader whose work can survive an examiner.
The engineering that closes that gap has a name: the harness. It’s everything wrapped around the model — the machinery that decides what the model is asked, what it’s allowed to touch, whether its answers are believed, and what gets written down. The picture to keep in mind is three boxes:
model → harness → bank
The model is rented capability. The harness is the product. This article walks through what’s actually inside Scout’s harness — not as marketing categories, but as the real mechanisms in the code, with the real numbers.
Asking well: the question is engineered before the model ever sees it
The first job of a harness is deciding what the model gets asked, because how you ask determines how reliably you’re answered.
Start with a single door. In Scout, no feature is allowed to call a model directly — every request, from every part of the product, funnels through one registry. That sounds like plumbing, but it’s the foundation everything else stands on: one place to enforce the rules, one place to record what happened, and no way for a new feature to quietly open a side channel.
Then there’s the question itself. Every reading request carries the same fixed instruction — the model is told it is “a precise document analyst” and that when asked to extract, it must “copy only literal, printed values — never derive, compute, judge, or invent.” That instruction never varies by mood or by user; it’s a constant in the code.
And the questions are deliberately small. Scout asks for at most eight facts per request. Not because more is impossible — because live testing showed that past roughly eight fields on a dense disclosure page, models start intermittently returning nothing for values that are plainly printed. The code’s own comment records the moment that number was chosen: the APR, the most standardized field on any disclosure, vanished from a twelve-field batch. The batch size is, in the code’s words, “sized against attention, not tokens.”
The same discipline applies to size. A request to the AI provider tops out at 32 megabytes, so Scout budgets under it on purpose: 20 megabytes for any single document, 26 for everything in a conversation combined — headroom deliberately reserved so no request can ever hit the ceiling mid-review. When a document is too big anyway, the harness shrinks it, splits it, or falls back to text, in a fixed order of preference. None of this is the model’s problem to solve. The harness solves it before the model is consulted.
Expecting failure: the model will flake, and the harness already knows
Models fail in undramatic ways: a timeout, a rate limit, an answer cut off mid-sentence, an empty response. A demo shrugs. A harness has a written policy for every one of these, because in production they happen daily.
When a call fails for a transient reason, Scout retries — up to four attempts total, waiting roughly one second, then two, then four, with a little randomness mixed in so a hundred workstations retrying at once don’t stampede the same server. Which failures are worth retrying is an explicit list in the code, not a guess: a server being momentarily overloaded, yes; a request the provider rejected as malformed, no — retrying that would just fail identically four times.
When a classification fails — the model can’t figure out what a document is — the harness escalates: the first pass runs on a fast, inexpensive model, and an empty or failed result gets one retry on a stronger, more expensive one. And if the stronger model can’t place it either, the harness keeps the honest answer, unclassified, rather than forcing a guess.
When an answer comes back cut off at the model’s output limit, the harness doesn’t retry the same doomed request in a loop. It applies a two-strike rule: the first truncation gets a notice and an adjusted approach; a second one ends the attempt with an explicit failure that says, in effect, ask for this in smaller pieces. A transcription that truncates gets split in half and re-run, halving again as needed, down to a single page.
The theme in all of it: failure is a normal input to the system, planned for in code, never a surprise handled by hope.
Not believing the answer: four checks before a value counts
Here is where a harness earns its keep. The model has read the page and returned, say, APR: 7.24%. A demo prints that on screen. Scout’s harness treats it as a claim to be tested, and runs it through a gauntlet.
Sanitation. Models leak formatting into answers — a real example from the code: a vehicle identification number returned with a stray comma glued to the end, an artifact of the model’s own output syntax. Edge characters like that get stripped; interior ones survive, so “CLEVELAND, OH 44104” keeps its comma.
Shape. A VIN must be 17 characters and can’t contain the letters I, O, or Q — that’s an international standard, not an opinion. A routing number carries a checksum. Scout keeps a fixed, closed vocabulary of these formats, implemented and tested in the engine itself, precisely so nobody authoring a review can accidentally write a sloppy pattern that rejects good values or accepts bad ones.
Type. A field declared as a number must actually be a number. A model answering “n/a” where a dollar amount belongs is recorded as a rejection, not passed downstream where it would quietly compare as false.
Corroboration. Finally, the harness checks — with plain text matching, no AI involved — that the extracted value literally appears in the document’s own transcription. This exists because of a real failure: a model once returned a VIN that passed every format check — 17 characters, legal charset — and the only thing that disagreed was the document itself, which never printed that value anywhere.
And there’s one failure no output format can prevent. Structured output — the industry’s standard technique for making models answer in machine-readable form — constrains the shape of a reply, never the contents of a string. Scout observed a live case where a model lost a closing quotation mark and produced one enormous value containing everything that should have followed it, with nine sibling fields silently reporting absent — all inside perfectly valid output. The harness now detects that signature specifically. That’s what a harness is: a place where every observed failure becomes a permanent check.
And when two documents disagree about the same fact — two different deductibles on two insurance documents — the harness doesn’t pick the more convincing one. It withholds the fact entirely and raises a conflict for a human, because, as the code puts it, picking one and reporting it as fact is the failure mode being fixed.
Fencing it in: what the model is allowed to touch
A model that can read your documents is useful. A model that can also delete things, call out to the internet, or rewrite its own instructions is a liability. So the harness grants capability the way a bank grants system access: explicitly, minimally, and audited.
An unattended review step in Scout gets exactly two tools: list documents, read document. Nothing else. There’s a test in the codebase that fails the build if that list ever grows — because it once did. The comment records the incident: the grant drifted to seventeen tools at one point, including the ability to publish and delete review logic, because two unrelated features shared a configuration. The fix wasn’t a policy memo; it was a test that makes the drift impossible to repeat silently.
The same posture runs through the whole product. When a model writes a database query, a guard checks it against a list of thirteen forbidden operations and requires it to be read-only — on top of a connection that was already opened read-only, so a violation produces a clear error naming the mistake instead of a confusing one. When the model generates a report or document for display, it renders inside a sealed sandbox that physically cannot make network connections — so a rendered page, in the code’s words, “cannot phone home.”
The model, in other words, operates inside the same assumption banks apply to everyone: capability is granted, not inherited.
Writing it all down: the ledger no call can escape
Remember the single door every model call walks through? It has a second purpose. Because every call passes through one registry, a recorder installed there captures every one of them — and, as the code states plainly, “no call site can add a new path that escapes metering.”
Every call is logged with the model used, what kind of work it was doing, tokens in and out, how long it took, and whether it failed — failures included deliberately, because a failed call still burned time, and counting failures is how you notice a problem’s tail. Those records land in the review’s own notebook file, so the audit trail travels with the evidence it describes.
Scout even audits the auditor. In test runs, the harness can deliberately plant known-wrong values — an OCR-style character confusion, a decimal shifted by a magnitude, a value stolen from a neighboring field, a plausible fabrication — to measure whether the verification step actually catches them. Because a reviewer that reads carefully and a reviewer that agrees with whatever it’s shown produce identical-looking records, and the only way to tell them apart is to lie to one on purpose and see if it notices.
The harness is the product
None of this shows up in a demo. A demo shows the model reading — the part that was already solved, the part every vendor’s product shares, because everyone is renting capability from the same handful of model providers.
What differs between products is the harness: whether questions are sized to what the model can reliably answer, whether failure has a plan, whether answers are tested before they’re believed, whether capability is fenced and audited, whether every call is on the record — and whether your documents traveled inside each request and were never stored, instead of being uploaded somewhere convenient.
So when you evaluate an AI product — this one or anyone’s — don’t ask whether the model is smart. They’re all smart. Ask what happens when it’s wrong, what it’s allowed to touch, and what got written down. You’re not buying the model. You’re buying the harness.