Measuring and Controlling Knowledge Interference in Enterprise Agents

The 30-second version

• We taught an agent a fixed set of verified expert corrections, then changed only how they were delivered. Force-fed into every prompt, they made the agent worse than the untaught model. Consulted on demand behind an applicability guard, they made it better. A tenth of the composite score, from delivery alone, crossing zero.
• Teaching moves its targets a long way (+0.86 on a case it governs) and taxes cases it was never about. We name this the pool tax, define it, and measure it replicating across seeds. A knowledge base is not a pile that only grows in value; it has a carrying cost that scales with how indiscriminately it is served.
• The guard's value tracks the base model's ability to judge relevance: it converts corrections into a gain on GPT-5.5 and only rescues GPT-4.1 from a collapse (0.135) back to parity. Guarded recall should get better as models improve.

Every team building enterprise agents eventually reaches the same step. The base model is competent but it does not know what your company knows. It has never seen your rate tables, your exception policies, your standing decisions about which supplier claims get disputed. So you teach it. You write down what your experts know and you put it in front of the model.

We ran that experiment carefully and found something we did not expect. The teaching helped or hurt depending entirely on how it was delivered, with the knowledge itself held fixed. Force-fed into every prompt, the same corrections made the system worse than the untaught baseline. Consulted on demand behind an applicability check, they made it better. The swing between the two was roughly a tenth of the composite score, larger than the gap between the taught system and the raw model in either direction.

We call the second pattern guarded recall, and this post covers the setup, the numbers, and what we think it means for anyone building agent memory.

Scope. Everything below is measured on phyvant-bench, a synthetic document world with a pinned regulation digest. It measures how learned knowledge should be delivered to an agent at run time. It does not measure recovering undocumented rules from real enterprise systems, which is a separate problem we report on separately.

The setup

The vertical is cross-border VAT and indirect-tax audit: 14 cases over a corpus of 19 documents, covering place-of-supply determinations, reverse charge, input-credit recovery, penalty computation, and cases where the only correct move is to abstain.

Fourteen cases is a small suite, and we designed the campaign around that fact: few cases, run many times, instrumented all the way down. Every number in this post is backed by a recorded trajectory, and the deltas that matter are reported per case and per seed rather than as averages.

The campaign behind this post
Cases per run	14, over 19 documents and 6 harness-owned tools
Scored runs	48 (delivery arms, seeds, correction rounds, executor runs)
Scored case-runs	758
Recorded tool calls scored	8,465
Model traffic	4,394 requests, roughly 41M tokens
Raw baselines	4 independent seeds (GPT-5.5), 3 (GPT-4.1)

Runs that failed their pre-run calibration probes are marked invalid in the ledger and excluded from every count and every number here. The bench itself is larger than this campaign: three verticals built on the same harness (VAT audit, royalty reconciliation, fund administration), 63 scored runs and over a thousand scored case-runs across them. This post reads out the VAT delivery experiment; the cross-vertical results ship separately.

Three design choices matter for reading the results.

The regulation digest is fictional and diverges from real law in places the gold answers depend on. A model that answers from memorized tax law instead of the pinned text walks into a trap. This is deliberate. In a real deployment the company's own policy is the authority, not the internet's average understanding of the rule, and we wanted a bench where that distinction is scored rather than assumed.

Scoring reads the recorded tool surface, never the model's narrative. Every case runs through a harness-owned set of tools: read a document, search the corpus, file a determination, post an adjustment, escalate, request a document. What gets scored is the write calls. Firing an adjustment on a case whose correct action is abstention is a scored failure no matter how well the model explains itself.

Verdict, action, and provenance are exact-match and citation-F1 with no model in the loop. The per-case composite is a fixed weighted sum:

$$S_i \;=\; 0.55\,S_i^{\mathrm{verdict}} \,+\, 0.18\,S_i^{\mathrm{action}} \,+\, 0.135\,S_i^{\mathrm{provenance}} \,+\, 0.135\,S_i^{\mathrm{rationale}},$$

and the first three components carry 87% of it with no model in the loop. Only the rationale consults a judge, its weight is capped at 13%, and the judge-free floor ships beside every number. A judge cannot carry a case.

The knowledge under test came from a correction round: a small set of learned facts derived from cases the system had previously gotten wrong. Two examples, paraphrased: that the tax point for a particular transaction class is the delivery date rather than the invoice date, and that a specific category of input credit is blocked.

Before anything was measured, every taught artifact was n-gram checked at $k=5$ against every gold answer. Zero of 14 cases showed contamination. We also ran a blank-context control, giving models the instructions and no documents, to confirm the case assignments themselves leak nothing: deterministic-match rates were 0.21 for GPT-4.1 and 0.29 for GPT-5.5, roughly what you would expect from guessing on that answer space.

The result

Two delivery mechanisms, identical knowledge, GPT-5.5 underneath.

Arm	Composite	Cost per suite
Knowledge force-fed into every prompt	0.396	$6.28
Raw model, taught nothing (4-seed mean)	0.443	$5.18
Knowledge consulted on demand behind an applicability guard	0.496	$8.14

Force-feeding cost 0.047 against the untaught baseline. Guarded recall gained 0.053. The delivery mechanism moved the system across a range of about 0.10, and it moved it across zero, which is the part worth sitting with. The same facts, correct in both arms, took the system in opposite directions.

Be careful about which part of that to believe at what strength. The raw model's four seeds span 0.394 to 0.481, a spread of 0.087, so the guarded run's margin over the best raw seed is 0.015, and a Wilcoxon signed-rank on the fourteen paired deltas does not separate either arm from raw at a single seed ($p = 0.77$ for the guarded arm): two large wins against several small losses is not a distribution fourteen cases can certify. The magnitudes are provisional point estimates and we intend to reseed both delivery arms. What we do stand behind is the direction and the structure: the swing is larger than the entire seed spread and systematic in sign, it reproduces on a second model, force-feeding's harm reproduces at five times the size on GPT-4.1, and the per-case pattern replicates across the correction round's two seeds. The sign is the claim; the second decimal place is not.

The two arms differ in exactly one term. Both build the model's context $C(x)$ for a case $x$ from the same taught pool $K$:

$$C(x) \;=\; x \,\cup\, \bigl\{\, e \in K \;:\; g(e, x) = 1 \,\bigr\}.$$

Force-feeding is the degenerate guard $g \equiv 1$: every entry, every case. Guarded recall makes $g$ an applicability check the model runs against an index of entry titles before the task, with full entries behind a lookup tool. Everything reported here is a measurement of how much that one predicate is worth.

The mechanism we believe is at work: knowledge that is always present is knowledge that is always relevant. A blocked-input-credit rule injected into a case about place of supply is not neutral context. It is a salient, confidently stated instruction competing for the model's attention against the case in front of it. Under a guard, the same rule stays silent until an applicability check fires.

The guard is not free. Consultations add turns, which is why the guarded arm ran about 1.6x the raw model's cost. At these deltas we consider that a good trade, but it is a real line item, and it is one more reason recall should fire only when a case actually needs it.

The pool tax

Composite scores hide where the movement came from, so here is the per-case view. First, the four cases the story is about:

Case	Before	After
Tax point is delivery date (taught)	0.00	0.86
Blocked input credit (taught)	0.31	0.95
Recovered from block modification (adjacent)	0.90	0.89
Caution misapplied (ungoverned)	0.87	0.27

Teaching moves its targets, and it moves them a long way. It also has a measured marginal cost on cases it was never about. The last row is a case that had nothing to do with either taught fact and lost 0.60 of its score.

Those four rows are only honest if the other ten cases held still, so here is every case, both delivery arms, against the same raw run:

The force-fed panel is the anatomy of the 0.047 loss: the always-present block fixed the one case its abstention rule governs (vat-009, +0.93) and flipped three high-scoring cases to near zero (vat-002, vat-006, vat-011), cases the injected rules had no business touching. The guard kept two of those three intact and confined the damage, while landing its gains exactly where taught rules govern.

We call this the pool tax: adding knowledge to a shared pool imposes a cost on cases that did not ask for it. Writing the paired per-case delta as $\Delta_i = S_i^{\mathrm{taught}} - S_i^{\mathrm{untaught}}$ and splitting the suite into the cases a taught entry governs ($G$) and the cases none does ($U$), the pool tax is the ungoverned side's average loss:

$$\tau_{\mathrm{pool}} \;=\; -\,\frac{1}{|U|} \sum_{i \in U} \Delta_i .$$

The partition is mechanical, not chosen. Every taught entry is authored with a stated condition, which the harness stores as the rule's scope of application, and a case belongs to $G$ exactly when some entry's stated condition matches the case's authored facts. Both artifacts, the entry conditions and the case definitions, exist and are versioned before any run is scored. This matters because a partition drawn after seeing which cases moved would make $\tau_{\mathrm{pool}}$ unfalsifiable: any regression could be retroactively declared governed and excused.

It is not a bug in the taught content. The content was correct and verified uncontaminated. It is a property of undifferentiated delivery: a delivery mechanism is only as good as its guard's precision on $U$, because that is where added knowledge has nothing to win and everything to leak.

We then ran a second correction round through the product surfaces, teaching a sweep methodology and explicit applicability boundaries for the existing rules, and scored it on two independent seeds. The composites came in at 0.522 and 0.449 against round zero's 0.496: averaged across seeds, the round produced no net gain. Read as an average, that is a correction loop stalling. The per-case deltas show something more specific:

The round worked. The sweep case (vat-013) recovered from zero in both seeds (+0.37, +0.33), which is the taught methodology engaging, replicated. And the round failed: $\tau_{\mathrm{pool}}$ came in at 0.068 on one seed and 0.112 on the other, a new boundary misfire cost the blocked-credit case about 0.65 in both seeds, and the tax-point recovery held in one seed and collapsed in the other. Both halves of the ledger are real, and they cancel. The composite went nowhere not because the teaching missed, but because the tax ate the gain.

That is the thesis demonstrating itself on our own correction loop, and it is the sharpest evidence in this post that the composite is the wrong instrument for evaluating one: a correction loop can be simultaneously working and net-zero. It also says the flat title index we used for $g$ is not enough as the pool grows; the guard has to get more precise with every entry added, which is exactly what the next round measures.

This reframes what a knowledge base for agents actually is. The naive model is a pile that only grows in value. The measured model is a pile with a carrying cost that scales with how indiscriminately it is served, and the carrying cost grows with the pool.

Where it did not replicate

On GPT-4.1, guarded recall scored 0.351 against a raw baseline of 0.358. A 0.007 deficit is well inside noise, so the honest read is parity: the guard did not help.

Force-feeding, for its part, was not a dent on the weaker model but a collapse: 0.135, down 0.223 from its baseline, nearly five times the damage it did to GPT-5.5. So the guard did do something on GPT-4.1: it rescued the system from catastrophe back to parity. It just could not convert the knowledge into a gain.

We think this is the most useful thing in the result and we want to state it plainly rather than bury it. An applicability guard asks the model to make a judgment before the task: is this piece of knowledge relevant here? That judgment is itself a capability. On a model strong enough to make the call reliably, the guard converts a liability into an asset. On a weaker one, the guard adds a decision the model gets wrong often enough to eat the benefit.

If that reading is right, guarded recall is not a universal fix. It is a technique whose value scales with the underlying model's ability to reason about relevance, which suggests it gets better rather than worse as models improve.

What we think this means

Three things we now believe and are building around.

Applicability is a first-class part of a knowledge artifact. Storing what your company knows is the easy half. Storing when it applies, in a form a system can check before recall, is the half that decides whether the knowledge is worth having. A correction captured without its scope is a liability waiting for the wrong case.

Knowledge accumulation should be measured, not assumed. If you are running a correction loop on an agent, the target case improving is not evidence the loop is working. The question is what happened to everything else. We would encourage anyone running this pattern to report paired per-case deltas across the full suite, not just the fixed case, and to do it across seeds.

Delivery is a design surface, not plumbing. In our runs it mattered more than the content. That is not a claim we expected to make, and it is the one we would most like other people to try to break.