Every Corobate receipt is sealed before anyone knows how the decision turned out. That one property is what makes it possible, months later, to score your own judgment honestly — and to make the next hundred decisions measurably better than the last hundred.
Imagine you are a weather forecaster. Every morning you write down "70% chance of rain" and seal it in an envelope. At the end of the year you open all 365 envelopes and check: on the days you said 70%, did it actually rain about 70% of the time? If yes, you are a good forecaster. If it rained 95% of those days, you were too cautious. If it rained 20% of the time, you were badly wrong. You cannot run that check unless the envelopes were sealed in advance. Corobate seals the envelopes. A backtest opens them.
Organizations do post-mortems all the time. They almost never improve anything, for four reasons that all have the same root cause: nothing was written down in a form that resists being edited by hindsight.
| The failure | What it sounds like in the room | Why it happens | What a sealed receipt does about it |
|---|---|---|---|
| Hindsight bias | "We all knew that supplier was shaky." | Memory rewrites itself after you learn the outcome. People sincerely believe they were more worried than they were. | The receipt says exactly how worried you were, in a number, sealed before anyone knew. Memory doesn't get a vote. |
| Survivorship | "Let's review the three deals that blew up." | Only failures get reviewed. The 200 decisions that went fine are invisible, so you can't tell whether your warning signs actually distinguish good from bad. | Every decision produces a receipt, not just the bad ones. The denominator exists. |
| Moved goalposts | "Well, we meant a different threshold." | The rule that was applied was never written down precisely, so it can be re-described after the fact to look correct. | The exact policy — every threshold, every required field, every weight — is sealed inside the receipt and can be re-executed by anyone. |
| Story over count | "Remember the Rotterdam thing." | One vivid disaster outweighs fifty quiet successes in human memory. | A backtest counts. Vividness has no weight in a contingency table. |
None of this requires anyone to be dishonest. Honest, careful, well-intentioned people do all four of these. That is precisely why the fix has to be structural rather than cultural.
Take every receipt issued in the review window — say six months. Not a sample, not the memorable ones. All of them. Each one already contains the inputs, their provenance classes, the reliability numbers, the gated confidence, the verdict, the policy that was applied, and the engine and weight versions that produced it.
For each decision, record the outcome: did the shipment pass inspection, did the regulator accept the documentation, did the supplier dispute the claim, was there a recall, a penalty, a rework. Crucially, the outcome is recorded as its own provenance-classed entry — a confirmed customs rejection is VERIFIED, a purchasing manager's recollection is SELLER-ASSERTED. You do not get to grade your predictions against evidence that is weaker than the predictions were.
For every field that ever appeared in a receipt — recycled content, cobalt provenance, state-of-health disclosure, certificate resolution, renewable share — ask one narrow question: when this field was weak, did things go badly more often than when it was strong? That is the field's predictive skill. Some fields will have a lot. Some will have none.
A field can be statistically associated with bad outcomes in the wrong direction — meaning the way you were using it was actively backwards. The backtest checks whether the relationship points the way the policy assumed it did. A strong relationship pointing the wrong way is a more urgent finding than no relationship at all.
If you test forty fields, a couple will look meaningful purely by chance. Section 5 explains the correction that removes them. This step is what separates a backtest from data-dredging.
Fields that earned their keep get promoted; fields that didn't get demoted or dropped. That produces a new numbered policy, e.g. weight_version: 2026.2. It applies to decisions made from that day forward. Every receipt issued under 2026.1 still re-verifies exactly as issued, forever. History is never restated.
The report is a receipt too. It records which hypotheses were tested, the counts behind each, the correction applied, and the resulting policy change. An auditor who wants to know why your thresholds changed in July has a sealed document that answers it.
Only three ideas do the real work here. None of them requires more than middle-school arithmetic to follow.
For one field, sort every decision into four boxes. Suppose the field is "state-of-health data disclosed."
| Outcome was bad | Outcome was fine | |
|---|---|---|
| Field was weak or missing | a — the field warned you, correctly | b — the field cried wolf |
| Field was strong | c — the field missed a real problem | d — the field was quiet, correctly |
A field is useful when the bad outcomes pile up in box a and the fine outcomes pile up in box d. A field is noise when the numbers spread evenly across all four boxes — which means knowing the field tells you nothing you didn't already know.
Suppose a field looked weak in 6 decisions and all 6 went badly. Six out of six is 100%, but six is not many. Corobate never reports the raw fraction. It reports a lower bound — the most pessimistic rate that is still consistent with what you observed, at 95% confidence. This is the Wilson score lower bound, and it is the same statistic Corobate uses everywhere else for the same reason: it refuses to let a small sample look impressive.
| What you saw | Raw rate | Reported lower bound | What that means in practice |
|---|---|---|---|
| 9 of 10 | 90% | 0.60 | Suggestive. Not enough to change a policy on. |
| 90 of 100 | 90% | 0.83 | Solid. Same raw rate, ten times the evidence. |
| 900 of 1,000 | 90% | 0.88 | Firm. Adding more data barely moves it now. |
Notice that all three rows have the identical raw rate of 90%, and the reported number climbs from 0.60 to 0.88 purely because the pile got bigger. That is the entire point. Confidence should come from evidence, not from arithmetic that happens to divide evenly.
Here is the trap, with a concrete number. Say you test 40 fields, and in truth none of them predicts anything at all. You use the usual "5% chance this is a fluke" cutoff. Then on average two of the 40 will still clear the bar by pure luck. If you promote those two into policy, you have just made your decisions worse while feeling like you made them better — and you will have a chart that supports you.
This is the single most common way a well-run analytics program quietly poisons itself, and it is why the correction in the next section is not optional.
Corobate applies the Benjamini–Hochberg false-discovery-rate procedure. The name is intimidating; the procedure is a sorting exercise you could do on paper.
Each test produces a p-value — the probability of seeing a pattern this strong if the field truly predicted nothing. Small p-value means "unlikely to be luck." Sort them smallest to largest and number them 1, 2, 3, … m.
Test number i out of m must beat (i ÷ m) × q, where q is the false-discovery rate you'll tolerate — commonly 0.10. So with 40 tests and q = 0.10, the strongest result must beat 0.0025, the second 0.005, the tenth 0.025, and so on. The bar is strictest for the first claim and loosens as evidence accumulates behind it.
Find the largest i whose p-value beats its bar. That one and everything above it are declared real. Everything below is discarded — including results that would have passed a naive 5% test.
This is the part that is easy to get wrong and that matters most. The denominator m is the lifetime hypothesis set, not the set you happened to run this time. Otherwise you could test five fields per quarter for two years and quietly accumulate the same forty tests, one small honest-looking batch at a time. Corobate keeps a sealed register of every hypothesis ever evaluated, so the correction cannot be diluted by slicing the work into batches.
Six months after the purchase order described in the exemplary scenario, Meridian's sourcing team runs its first backtest. The window covers 46 sealed decisions on battery-module procurement. Of those, 17 were held for evidence — a 37% hold rate — which the team suspects is too high but has never been able to prove either way.
| Quantity | Value | Note |
|---|---|---|
| Decisions with sealed receipts | 46 | Every decision in the window, not a sample |
| Held for evidence (WITHHELD or CAUTION) | 17 (37%) | Each hold cost roughly $18,000 in carry and expediting |
| Adverse outcomes recorded | 9 | Disputes, rejections, rework or penalty events |
| Outcomes classed VERIFIED | 7 of 9 | Two rested on internal recollection and were classed SELLER-ASSERTED |
| Distinct fields tested | 11 | Added to a lifetime register that now holds 11 hypotheses |
| False-discovery rate applied | q = 0.10 | Benjamini–Hochberg across all 11 |
| Field | What the counts showed | Survives FDR? | Action taken |
|---|---|---|---|
| State-of-health disclosure soh_algorithm_disclosure |
Absent in all 6 of the 6 decisions that later produced a warranty or performance dispute. Present in every decision that did not. | Yes — clears its bar comfortably | Promoted. Reliability requirement raised: this field must now be VERIFIED, not merely present, before a release can reach APPROVE. |
| Certificate register resolution certificate_register_resolution |
All 3 decisions where a cited certificate number failed to resolve in the issuing register produced a downstream documentation dispute. 3 for 3. | Yes | Promoted to a mandatory pre-check. Any certificate identifier appearing in an input is now resolved against the issuing register before the receipt is built. An unresolvable identifier is recorded as a named coverage gap rather than passing through silently. |
| Renewable energy share renewable_energy_share |
Missing or weak in 11 of the 17 holds — the single largest driver of holds in the window. But the 11 held decisions showed no relationship to any adverse outcome once released. | No — fails the correction | Demoted. Removed from the sufficiency-critical set for this decision type. Still recorded on the receipt as disclosed context; no longer gates a release. |
| Cobalt provenance cobalt_provenance |
Sat at 0.72 (MODELED, book-and-claim) in 14 decisions that reached CAUTION. Zero of the 14 produced an adverse outcome — but zero is also consistent with a low base rate over a short window. | Inconclusive — insufficient power | No change. Explicitly flagged for re-test at the next window rather than quietly dropped. Absence of evidence is reported as absence of evidence. |
| Seven other fields | No signal, or too few observations to test. | — | Retained unchanged; carried forward in the lifetime hypothesis register so future corrections count them. |
17 of 46 down to roughly 6 of 46 when the new policy is applied to the same decision mix. The drop comes almost entirely from retiring the renewable-share gate, which was generating holds that predicted nothing.
Eleven fewer holds at about $18,000 of carry, expediting and administrative cost each, over a comparable half-year window — with detection of genuine problems unchanged or improved.
The important thing about that result is what it is not. It is not "we decided to be less cautious." Two gates got stricter — state-of-health now demands verification rather than mere presence, and certificate numbers must resolve against a register before they count. Caution was reallocated, not reduced: moved off a field that was producing expensive false alarms and onto two fields that had a demonstrated relationship to real losses. A team that had simply relaxed thresholds to hit a hold-rate target would have gotten the same headline number and a worse business.
A learning system that can revise its own past is not a learning system; it is a story. Corobate draws a hard line.
| Can be changed by a backtest | Can never be changed |
|---|---|
| Which fields are sufficiency-critical for a decision type | Any receipt already issued — its content, hash, chain position or timestamp |
| The reliability floor a field must reach (e.g. present → VERIFIED) | The verdict that was reached under the policy in force at the time |
| Which pre-checks run before a receipt is built | The recorded weight_version and engine_version of a past receipt |
| Approve/caution thresholds for future decisions | The recorded outcome, once sealed with its own provenance class |
| The default provenance tier applied to a source that has proven better or worse than assumed | The direction of the chain — new entries only ever append |
Concretely: a receipt issued in March under weight_version 2026.1 re-verifies in December, and in five years, exactly as it did the day it was sealed. A verifier re-runs the gate using the version recorded in that receipt, not the version currently in force. If the policy has since changed, the receipt is still correct — it was correct under the rules that applied, which is the only sense in which a past decision can be correct at all.
This is also the answer to the obvious auditor question, "couldn't you just tune the policy until the receipts look good?" No — because tuning applies forward only, the old receipts don't move, and the backtest report that authorized the change is itself sealed and inspectable. You can see what was changed, when, on what evidence, and what the evidence bar was.
Where an engine adjusts its own defaults from backtest results, the update is gated by a sealed baseline manifest: the new configuration must reproduce a fixed set of known-answer test cases before it can take effect. An update that would change the answer to a case whose answer is already established is refused. This keeps a recalibration from silently drifting the engine's behavior on cases nobody re-checked.
Six questions to ask of any backtest report — including one produced by Corobate. A report that cannot answer all six is telling you a story rather than showing you a scoreboard.
| # | Ask | What a good answer looks like |
|---|---|---|
| 1 | Were the predictions sealed before the outcomes were known? | Every receipt carries a timestamp token over the chain head that precedes the outcome record. Not "we wrote it down somewhere." Ask the follow-up too: whose authority issued it. A token from the same party that wrote the record proves sequence to that party and to nobody else. Ours today is a demonstration authority under a root we ship ourselves; pointing it at a public one is the step that turns this row from a mechanism into evidence. |
| 2 | Does the denominator include the decisions that went fine? | All 46, not the 9 that went wrong. If only failures were reviewed, the report cannot tell you anything about false alarms. |
| 3 | How strong is the evidence for the outcomes themselves? | Outcomes carry provenance classes. Seven VERIFIED, two SELLER-ASSERTED, stated openly — not a single undifferentiated "9 failures." |
| 4 | How many hypotheses were tested, in total, ever? | A specific number, drawn from a lifetime register, used as the denominator in the correction. Not "the ones we looked at this quarter." |
| 5 | Did anything get stricter? | At least some findings should tighten requirements. A backtest that only ever loosens things is a budget exercise wearing a lab coat. |
| 6 | Do the old receipts still verify? | Re-run the verifier on a receipt from before the change. It should pass, unmodified, under its own recorded version. |
The sixth question is the one an auditor will actually run. It takes about a second and it is the whole argument: if last spring's receipts still verify byte-for-byte after the policy changed, then the policy change didn't touch history, and the record of your diligence at the moment of decision is exactly what it was.
The recalibration loop is the subject of its own independent claim. Plain-language summaries below; the operative language is in the application itself.
| Claim | What it covers | In plain terms |
|---|---|---|
| 38 independent |
The recalibration method: sealing attestation records prior to outcome availability, later associating outcome data, computing per-feature predictive performance, and deriving an updated weighting applied prospectively. | The whole loop described on this page — seal first, score later, update forward. |
| 39 | Controlling the false-discovery rate across the evaluated hypothesis set using a Benjamini–Hochberg procedure. | Section 5. The rule that stops noise being promoted into policy. |
| 40 | Point-in-time constraint on the evaluation: inputs observed after the applicable period end are excluded. | You cannot score a decision using information that did not exist when it was made. No accidental time travel. |
| 41 | Anchoring of the sealed records such that the pre-outcome sealing time is independently evidenced. | A third party can confirm the envelope really was sealed before the result came in. |
| 9 · 35 | Engine and weighting version identifiers sealed into each record. | Every receipt knows which rulebook produced it, so it can be re-checked against that rulebook and no other. |
| 17 · 18 | Self-update gated by a sealed baseline manifest of known-answer cases. | A recalibration that would change an already-established answer is refused before it can take effect. |
| 4 · 26 | Wilson score lower bound on observed proportions. | Small samples are reported pessimistically. Six-for-six does not get to look like certainty. |
| 5 · 23 · 33 | Named abstention as a first-class outcome with recorded reasons. | "We don't know yet, and here is precisely what we're missing" is a legitimate result the system is designed to produce. |
Continue: Nuts and Bolts — how a receipt gets built · Exemplary Scenario — the decision this backtest reviews · For Executives — what the improvement is worth in dollars.