The 30-second version
- We built a system that watches work happen, compiles the proven parts into deterministic execution, and spends the model only on what remains genuine judgment. On three judgment-heavy back-office environments it outscores the frontier model it runs on: 0.60 vs 0.46 on VAT audit, 0.71 vs 0.56 on fund administration, 0.61 vs 0.49 on royalty reconciliation.
- The gap is widest where it matters most: reliability and cost. The compiled system retains 89 to 95 percent of what it passes across four independent reruns, at $0.04 to $0.06 per case. The raw frontier model runs 3 to 4x the cost with retention as low as 0.77.
- Every claim was preregistered before its runs, thresholds never moved, and the misses are recorded verbatim. The negative results shaped the system: unproven defaults misroute minority cases, taught knowledge taxes what it does not govern, and a judgment loop that stops at the first write call silently destroys correct abstentions.
There is a default bet in enterprise AI right now: models are getting better, so agents will get reliable on their own. Wait a generation, prompt harder, retry more.
We took the opposite bet. Most back-office work is not an open-ended reasoning problem. It is procedure plus a small number of genuine judgment calls, performed by people who never wrote the procedure down. If a system can watch that work, compile the parts it can prove into mechanical execution, and reserve the model for the judgment that is actually left, then reliability stops being a property you hope the next model has and becomes a property you accumulate.
This post reports what happened when we measured that bet properly: three adversarially hardened environments, five systems, every cell at four or more independent reruns, every claim registered before the run that tested it. The compiled system beat the strongest available frontier model on all three environments while running on that same model as its engine. The engine did not get smarter. The system around it earned the difference.
Scope. Everything below is measured on phyvant-bench, our suite of synthetic document worlds with pinned regulation digests, fictional jurisdictions, and oracle-verified gold. It measures whether observed work can be compiled into reliable execution. It does not measure integration with real enterprise systems, and the environments are fictional by design, which is what makes the contamination controls possible. How taught knowledge should be delivered to the model at run time is its own study, reported in guarded recall.
The environments
Three verticals, each a full document world rather than a question set: cross-border VAT audit, private-fund administration, and licensing royalty reconciliation. Each has its own fictional jurisdiction whose pinned regulation digest deliberately diverges from real-world rules, a data room of ledgers and contracts and correspondence, findings-mode sweeps, cases whose only correct answer is to abstain, and adversarial distractors. The agent reads the record through tools, resolves contested evidence, applies the pinned regulation, and files a determination with amounts. Scoring reads the recorded tool ledger, never the model's narrative.
| The campaign behind this post | |
|---|---|
| Environments | 3 (VAT audit, fund administration, royalty reconciliation) |
| Cases per suite | 14 / 10 / 14, over 19 / 21 / 16 documents |
| Scored suite runs in the data lake | 177 |
| Scored trajectories | 2,306 |
| Recorded tool calls scored | 24,567 |
| Seeds per reported cell | 4 or more, uniformly |
| Oracle solver | 1.000 through the same tools agents get |
Hardness is a maintained property, not an accident. Frontier scores on VAT fell from 0.82 to roughly 0.46 across six de-leaking rounds while the oracle held at 1.000, which is the difference between a benchmark that is hard and one that is broken. Twice, a hardness probe turned out to be a gold audit: in fund administration, the original gold on one case rewarded a naive weekday count. The holiday-aware count, which the pinned agreement requires, gave nine business days against a ten-day requirement. Fixing the gold created a trap that every frontier model now fails at 0.000, while the compiled system files it correctly from a mined policy.
The compile ladder
The system is a ladder of mechanisms, each of which converts a class of model behavior into deterministic execution once, and only once, it has been proven from observation.
Locked walks. The app's real observation-to-mining pipeline compiles demonstrated work into procedures whose read steps execute with zero model calls. Two observations of a demonstrated flow are enough to reach the structural lock ceiling.
Decision locks with proof floors. Declared decisions mint policies with per-guard support and confidence, plus a separate proof over the unguarded residual. The executor commits a decision without the model only past those floors, and only inside the distribution the policy was mined from. The in-distribution gate exists because we measured what happens without it: a fully proven default absorbed every novel case it had never seen, including the ones whose correct answer was to abstain.
Shape-aware commit. Per-vertical performance differences turned out not to be noise. They decomposed exactly by episode shape. Substantive determinations pin a single filing call. Enumeration episodes, scored on findings F1, keep full judgment with only the determination pinned. Resolved abstentions ride into the judgment loop as stated precedent, never forced. This one change flipped fund administration from losing to the raw model by 0.10 to beating it by 0.15, with zero per-vertical configuration.
Mined arithmetic. The miner searches demonstration evidence for formulas that reproduce the filed amounts over a closed operation vocabulary, proves each candidate by support and by backtest through the same evaluator that will run it live, and re-fetches its source documents raw at execution time. It discovered 187,500 x 0.21 and its siblings unprompted. The practical effect: the amounts that used to be re-derived by the model on every run, differently, became compiled arithmetic. VAT retention went from 0.667 to 1.000, twenty out of twenty mechanical amounts exactly equal to gold.
What remains after the ladder is genuine judgment, and the model gets exactly that.
The result
Five systems, identical suites, identical scoring: the compiled system running natively in the product, the frontier model with taught organizational knowledge, the frontier model alone, and the same pair for the widely deployed model class. All cells at four or more seeds.
| environment | Phyvant (compiled, native) | GPT-5.5 + taught skills | GPT-5.5 | GPT-4.1 + taught skills | GPT-4.1 |
|---|---|---|---|---|---|
| VAT audit | 0.6038 | 0.4780 | 0.4645 | 0.3042 | 0.3080 |
| Fund administration | 0.7068 | 0.5312 | 0.5626 | 0.2721 | 0.3190 |
| Royalty reconciliation | 0.6066 | 0.4619 | 0.4917 | 0.3364 | 0.4302 |
The composite blends verdict correctness with process quality. But the composite is the least demanding lens. The two that decide whether you can deploy are reliability and cost:
| environment | retention (pass^4 / pass^1) | cost per case |
|---|---|---|
| VAT audit | 0.952 vs 0.909 raw | $0.042 vs $0.163 raw |
| Fund administration | 0.889 vs 0.833 raw | $0.054 vs $0.120 raw |
| Royalty reconciliation | 0.941 vs 0.769 raw | $0.064 vs $0.092 raw |
Retention asks: of the cases you solve at least once, what fraction do you solve every single time across four independent reruns? A case that passes three runs out of four is a case you cannot put in production. The compiled tier's best configuration on VAT holds retention at 1.000, because a locked read, a committed decision, and a mined formula do not have moods.
And to be precise about what "beats the frontier" means here: the compiled system runs on GPT-5.5. Same engine, same tools, same suites. The comparison is engine-controlled by construction, which removes the usual escape hatch that the winner simply used a better model.
The negative results are the mechanism
Every rung of the ladder exists because something failed in a measured way first. Three of those failures changed our architecture; we think they generalize to anyone building agent systems.
Taught knowledge taxes what it does not govern. The taught-skills arms exist so the frontier baselines get the same organizational knowledge the compiled system mines from. Measured at full seed discipline, skills land at or below the raw model on every one of the six non-VAT cells, for both models. Teaching moves its targets and taxes everything else, and smarter retrieval amplified the tax rather than fixing it. The road to accuracy is not a bigger knowledge pool. It is the deterministic tier, with knowledge consulted narrowly where it applies. The full delivery study is in guarded recall.
Proof is distribution-bound. A decision default proven beyond every statistical floor still absorbed every novel case on a held-out world, including the abstention cases. Support and confidence tell you a policy held where you watched it. They tell you nothing about where you did not. Production consequence: every policy carries its mined feature vocabulary, and defaults fail closed outside it.
Judgment loops must never terminate on a write. A compound abstention files a request for the missing document and then files cannot-determine. A loop that stops at the first write call destroys the second action, and the scored diff was brutal: one fund-administration abstention case went from 0.000 to 0.904 in both seeds once the loop was allowed to finish. We found this because per-case parity between two implementations of the same runtime is part of our regression suite, and eight of ten cases matched exactly while the two abstention cases did not.
There are seven more of these, and they live with their evidence trails in the repository. The discipline that produced them is the part we would defend hardest: thresholds written and committed before any run, thresholds never moved, misses recorded verbatim in the registration document next to the claims they missed.
Why this matters
The industry's reliability roadmap is mostly "the next model." Our data says something more specific: on judgment-heavy procedural work, the distance between a frontier model and a deployable system is not model quality. The same engine that scores 0.56 alone scores 0.71 inside the compiled system, at a third of the cost, with better retention, and the difference was built from observation, proof floors, and the humility to let measured failures redesign the architecture.
Determinism is not something you declare. It is something you earn, one proven mechanism at a time, and the model should be spent only on the judgment you have not earned yet.
Every number in this post resolves to a run artifact. The benchmark, the registration document with its verbatim misses, and the full research record are in the phyvant-bench repository.