31 Of 36 Fields Go To A Human. That Is The Feature.
A customs broker receives the paperwork for one shipment: a bill of lading, a commercial invoice, a packing list, a certificate of origin, a customs declaration. Scans of scans. Three languages. A stamp printed across the container number. A correction in biro in the margin.
Every document AI system built for this eventually reaches the same question, and nearly all of them answer it the same way: at what confidence do we publish?
Somebody picks 0.95. It sounds careful. Nothing supports it.
That number is the most dangerous artefact in the whole system, because it looks like a measurement and it is a preference. It was chosen in a meeting, it is never revisited, and every downstream figure inherits its authority without inheriting any evidence.
Manifest refuses to answer that question. Everything below is what happens when you take the refusal seriously.
What it is
A document intelligence platform for cross-border trade, on AWS. Documents land in S3, a Step Functions state machine reads them through a cost-ordered cascade, and three things come out: a structured record per document, agreement (or disagreement) between documents that must agree, and a tariff classification.
Textract · Bedrock · Bedrock Data Automation · SageMaker · Step Functions · Lambda · Iceberg on S3 · Athena · OpenSearch · Redshift · EMR Serverless · Terraform.
The design rule underneath all of it is a boundary. Models read; deterministic code decides. A model may read characters off a degraded scan, propose which field a token belongs to, and propose a tariff code. It may never decide whether a value is confident enough to publish, whether the value actually appears where its provenance says it does, whether two documents agree, or whether a human's decision was recorded. There is a CI check that fails if a cloud SDK, a clock, or an engine's name appears anywhere in the decision core.
The threshold nobody can justify
Instead of a confidence threshold, each field declares an error budget: the rate of published-and-wrong the business is willing to carry on that field. container_number carries 0.001. buyer carries 0.01.
The threshold is then derived from that budget — the lowest score whose upper confidence bound on the published-and-wrong rate still fits the budget, against a labelled set, with N printed beside it.
And where no threshold fits the budget at the data available, the field does not quietly get a lower one. It is declared always-review.
gross_weight lands at 0.910 against a 0.005 budget, with a bound of 0.004, publishing 753 of 989 at ECE 0.07. Thirty-one do not, and each says why.The reasons matter more than the count, because they point at different fixes:
- Evidence-limited — zero errors in the high-confidence population, but a 95% upper bound still wider than the budget.
date_of_issueis 0 wrong out of 468 above 0.9 and would need n=1,497 at zero errors to prove it. The reader is not wrong; there is not enough labelled data to prove it right. The fix is more data or an escalation tier, not a different number. - Quality-limited — real errors survive at high confidence. That is over-confidence, not thin evidence, and it is what the cascade exists for.
This is the difference between a system that abstains and a system that is honest about why it abstains. And it is scored against the alternatives rather than asserted: publishing everything is 26.72% wrong; a hand-picked 0.85 is 4.68%; the derived policy is 0.22%. The comparison that matters more than the rates is the one underneath them — publishing everything misses 34 fields' declared budgets, the hand-picked 0.85 misses 28, and the derived policy misses none.
One more thing keeps it from being a photograph. The thresholds are derived from a committed engine recording, not from a live run — because the reader is a binary, and a binary produces different confidences on a different build. Regenerating that recording prints the movement of every threshold, per field, old against new, and refuses to overwrite until the shift is explicitly accepted. An engine upgrade that quietly moves a threshold nobody looked at is the whole claim becoming decoration.
Every field traces to a pixel
The second rule: a published field that cannot be located on its own page is a build failure.
incoterm published as "DAP" at 0.970, on page 1, at box [0.4962 0.2357 0.0250 0.0077] — and the crop those coordinates point at, enlarged, on the right.Three checks stand behind that, and the article's honesty depends on saying they are not equally strong:
- The crop carries ink where the record says a value is. A box pointing at white space fails this whatever produced it, and it is independent of every reader.
- A re-read of the crop agrees with the published value. This is an independent path, not an independent engine — it catches a wrong box and it cannot catch a confident misread. Saying otherwise would be the easiest lie in the project.
- A check-digit field that disagrees with its own arithmetic is refused outright. Container numbers have a check digit. No model is involved in that refusal at all.
On a sample of 120 records: 94 of 120 honest records verified. Then the same gate was attacked with corrupted geometry, and it refused 120/120, 120/120 and 6/6 — each by the layer that should have caught it.
Doctrine rule seven is the door with no key: a field with no provenance cannot be overridden into existence. Not by a model, not by a pipeline, and not by an approver — because if the system cannot point at where a value came from on the page, nobody has the information the approval would be about.
The documents must agree, and the disagreement is the signal
One shipment arrives as five documents written by different parties, and the interesting failure is not that one of them is unreadable. It is that two of them are readable and say different things. A gross weight of 2,450 kg on the bill of lading and 2,540 kg on the packing list is a transposition somebody will otherwise notice at a border.
Which fields must agree across which document types, with what tolerance and in what unit, is declared in a contract — data, not code.
Two things keep that zero from being decoration. The generator that plants a disagreement is blind to the reconciliation contract — it perturbs a value without consulting the rule that will catch it — and the expected findings are derived from ground truth by a separate path. A planter that reads the contract to decide what to break and a detector that reads the same contract to find it is one function agreeing with itself.
And 407 pairs are reported not comparable, because one side abstained. An abstention is not an agreement, and it is never counted as one.
What the abstention costs, and who pays it
Here is where most human-in-the-loop systems stop, and it is exactly where the interesting part starts.
Abstention is the safe answer, and abstention is not free. Send 40% of fields to a human and you have not built a careful system. You have built a queue nobody can staff — and the reviewers will start approving without looking.
So the review queue is a declared finite resource, and exceeding it fails the build.
It goes green only because a named person accepted the overage, in a file, with an expiry date on it — ACCEPTED tier_zero_only_extraction by Theofanis Tsakanikas, expires 2027-02-09 — and the acceptance states its own cause and its own permitted response.
The response that is forbidden is the obvious one. Raising a confidence threshold to reduce queue volume, without changing the error budget, inverts the derivation and turns a derived number back into a chosen one. There is a planted mutation in the build that attempts precisely that, and it must be refused by name.
Reviewer corrections feed back, because a correction is the highest-quality label this system will ever see — a human looked at the page and said what it said. But the arrow that does not exist is the one that matters: corrections move N, never an error budget. A budget that relaxed under queue pressure would be the forbidden move wearing a feedback loop.
A human decision is only evidence if the human was looking
A reviewer who agrees with the model every time is not a control. They are a rubber stamp with a login.
So time on task, agreement rate and sampled re-review are measured, and the pathologies are named rather than averaged away:
- reviewer-2: 400 decisions, 100.0% agreement, median 2 seconds. Reported as a rubber stamp. An approval from a reviewer who agrees with everything is excluded from the feedback loop and counted, because admitting it would lower the observed error rate using evidence nobody produced by looking at anything.
- reviewer-3: 150 decisions, 0.0% agreement, median 30s. The same finding wearing the opposite sign: a reviewer who never agrees is not reading either, and that is the tail nobody alerts on.
- reviewer-1: 240 decisions, 83.3% agreement, median 24s. The one that looks like work.
agreed_with_model and seconds_on_task are columns, not a report — which is what makes reviewer integrity a query rather than a promise.And a correction never erases what was published. A decision writes a new version that supersedes the old one; both stay retrievable, with the diff.
The same rule reaches the far end of the warehouse, where a wrong answer is money. A tariff classification is genuinely ambiguous across thousands of classes, and a wrong one is quantifiable financial and legal exposure — so the classifier proposes candidates and returns decided: false, and the duty line only exists once a person has approved it:
WHERE NOT human_decided returns 0. Doctrine rule five — nothing approves itself — as a count against the mart rather than as a paragraph in a design document.Every gate is attacked, on every push
A gate nobody has tried to break is a gate nobody has tested.
make gate-proof copies the repository, plants a real violation, and fails unless the named gate refuses it for the right reason. Three rules keep it honest: the baseline must be green first; a non-zero exit is not evidence on its own; and a mutation whose target has moved is reported STALE, never quietly passed.
let a tier that reports no confidence publish, let review volume relax the error budget, count an approval from a reviewer who agrees with everything, let a blank crop verify, let a container number skip its own arithmetic.Several of these exist because the mutation was accepted the first time it ran. One injection rule fired on "Please disregard the previous packing list" — which is an ordinary amendment note, printed on real paperwork every day. The rule was anchored on the wrong half of the sentence, and gate-proof is what found it. An over-broad control is not a safe control; it is a control somebody switches off.
What actually ran
None of this is a slide deck.
VerifyProvenance sits between the threshold and the publish, and QueueForReview is a terminal state rather than an error path.One dispatch ran the whole suite, then applied five Terraform layers — foundation, lakehouse, extraction, batch, analytics — in 49 minutes 32 seconds, with Textract, Bedrock, SageMaker Serverless Inference, OpenSearch Serverless and Redshift Serverless switched on. Then documents went through the deployed pipeline and the estate was asked whether it had behaved.
Then it was destroyed the same day. The resting state of the repository is a state bucket, its key, five parameters and a deploy role. The teardown is the half of the pair that usually gets written and never run; this one runs, and a sweep script exits non-zero if anything survives.
And every claim is still scored offline — 527 tests, 15 harnesses and those 61 planted violations, on a laptop, in twelve minutes. That is deliberate: a claim that needs a running estate to check is a claim nobody can reproduce.
An abstention is a row, not a gap
The quiet failure mode of every thresholding system is that the analytics layer only sees what published.
And because they are rows, claim 1 stops being a harness result and becomes a query anybody with the account can run. Add one line to that statement — AND confidence < threshold — and the answer is the only one that is allowed:
Results (0). Not a test asserting it, not a fixture: the published set in the lake, asked directly whether anything in it cleared a threshold it did not meet.Missing is missing, and it is stated. No field is ever filled with a modal value, a zero, or "the usual" — a default is a lie with a plausible shape.
The cost is a model, and the column says so
This is the line I hold hardest, because it is the easiest one to cross by accident.
modelled_cost, and the currency column is called modelled_currency. Not documentation — the schema refusing to let a modelled number be reported as a measured one.Exactly one euro figure in this project is measured: $3.50, Textract DetectDocumentText over 2,336 eligible corpus pages on 2026-08-15, at the published per-page rate. It may be quoted with its date.
Everything else is a model: routing measured over 36,078 recorded pages, multiplied by published unit prices, giving 0.59 USD per 1,000 pages — with tier 3 explicitly not priced, because it has barely been called.
Which brings me to the sentence this project is not allowed to say. "Accuracy held at X for Y% of the cost" is the standard cascade claim, and it is unavailable here. The upper tiers have been called, not measured. What the cascade eval proves is two things and then stops: the routing rule sends the low-confidence pages up, and the pages it keeps at tier 0 meet their fields' error budgets. The value of the escalated fraction is an assumption, it is labelled one, and the sensitivity of the cost model to it is shown rather than hidden.
What I'm not claiming
The corpus is generated. Every confidence in the first two claims comes from a real reader on generated paper, and the only answer to "did you tune the generator until the claims passed?" that does not come from the generator's author is a set of 100 genuinely photographed documents nobody here designed — where calibration error is 0.0592, against 0.0815 on this repository's own corpus. The reader is, if anything, better calibrated on paper nobody here designed — a difference of 0.0222, inside a declared tolerance of 0.15. Accuracy is deliberately not compared: photographs of thermal-printed receipts against rendered trade documents is a difficulty comparison, not a calibration one. The generator also declares its own operating envelope in a committed file, and a test goes red when it drifts out of the band, in either direction.
The honest-limits section of the README is longer than most projects' feature lists, and it includes defects found by querying the live estate rather than by running a test. The best of them: a document that abstains on every field never reaches the lake — its branch ends at QueueForReview with no landing step — so the analytics layer undercounts abstention by exactly the hardest documents.
The defect, in the console that revealed it. AnythingPublishable takes the default branch, the abstentions are written and queued, and the run ends — LandInTheLake is on the other path. Nothing failed, which is why no test caught it. Also named there: no distributed job has ever executed, the reader image carries known base-image CVEs that no gate reads, and everything is one region, one run, one day.
The takeaway
A system that publishes everything is not confident. It is unfalsifiable.
The capability here is the refusal, and what makes the refusal trustworthy is everything around it: a threshold derived from a declared error budget rather than chosen in a meeting, a box checked against the page rather than against the record, a review queue with a capacity that the build enforces, reviewers graded on whether they were actually looking, and 61 planted violations that must each be refused by name.
Take any one of those away and you do not get a slightly less careful system. You get a system that abstains for reasons nobody can inspect — which, from the outside, is indistinguishable from one that publishes for reasons nobody can inspect.
Full repo — the derivations with their N, the provenance gate, the review harness, the live-run screenshots and the limits written next to the claims: https://github.com/theofanis-tsakanikas/manifest
If you have shipped a system that had to say "we cannot state that": what did it take to make abstaining acceptable to the people paying for it? That is the part I found hardest, and I am collecting answers.
One of a series of write-ups on the projects in my portfolio — each one a reference implementation of the trust layer that makes data and AI safe to ship.
The code, the CI and every test behind this article are public.