The Provenance Gate: Why an AI Agent Can't Write a Fact Without a Source
By Michael Oskola, Founder & AI Automation Architect · September 17, 2026 · 8 min read
A provenance gate is a rule that sits in front of an AI agent's memory: an agent can propose a fact, but it can't become memory until it names where the fact came from. No source, no write. We built one for a hackathon project called FleetMemory, then attacked it ourselves. The first version let a whole class of lies through, 6 out of 6 tries. The fixed version blocked 30 out of 30.
What a provenance gate actually checks
Most agent memory systems trust whatever the agent writes. That's the failure mode: an agent infers a number, states it with confidence, and the system files it next to facts a customer actually said. Read it back in a month and there's no way to tell which is which.
A provenance gate adds one deterministic check before any write commits: does this fact trace to a source? A customer saying "our budget is now $14,000" is sourced, and it supersedes the old value. An agent inferring a budget from context, with nothing a person actually said behind it, is not sourced. That claim quarantines instead of writing straight to memory.
In the FleetMemory demo, a rogue agent asserts a budget of $1,800 through the exact same write path everything else uses. The gate can't tell that number is false; it isn't comparing values, it's checking for a citation. It quarantines the claim with a gold "held for review" stamp, and an adversarial verifier reads the case and rejects it, right above the real fact it contradicts: $18,000, sourced to an actual customer message.
(opens the full-size image in a new tab)- Junk and duplicate checks: malformed or already-known writes drop before they cost a verifier call.
- Contradiction checks: a new value that conflicts with an existing fact gets flagged, not silently overwritten.
- Confidence floor: a guess below a minimum confidence never reaches memory.
- Provenance check: a claim with no customer utterance behind it quarantines, no matter how confident the agent sounds.
The red-team run that found the hole
A gate is only as good as what you throw at it. We ran 30 adversarial writes across 5 attack classes against the real gate and verifier, not a mock. The script and full results ship in the public repo.
The first run caught a real gap: unsourced novel claims, facts that don't contradict anything already in memory because there's nothing to contradict, sailed past the deterministic gate 6 out of 6 times. A brand-new, confident, entirely made-up fact walked straight into the fleet's shared memory.
The fix was a rule we call needs_provenance: a claim with no customer utterance behind it quarantines and faces the verifier, and claimed confidence alone can no longer auto-supersede a fact that already has a source. We re-ran the same 30 attacks. This time all 30 were blocked, including junk, oversized payloads, absurd-confidence guesses, unsourced contradictions, and the unsourced novel claims that got through the first time.
The gate still let through 9 of 10 legitimate customer-confirmed updates, not 10 of 10. The one it missed is a disclosed race condition: a value looping back to what it originally was while an intermediate update sat unresolved. It's traced in the decision journal, not hidden or tuned away. Median gate latency across the run: 293 milliseconds.
(opens the full-size image in a new tab)Why the verifier defaults to reject
A quarantined claim doesn't just sit there. An adversarial LLM verifier reads it and rules on it, and its default is reject, not accept. If the verifier call itself fails for any reason, that also counts as a rejection.
The verifier's job is to weigh provenance specifically: a direct customer statement supersedes an old fact; an unsourced value gets rejected, even if it sounds plausible, even if the agent stating it sounds confident. Every verdict, with a one-sentence reason, gets written back onto the same decision-journal row as the original write attempt.
Corrections don't delete anything either. A fact that gets superseded keeps its old row, marked invalid and pointed at whatever replaced it. Ask the system what it believed on any past date and it can tell you, including beliefs it has since abandoned.
(opens the full-size image in a new tab)The strongest kind of proof: someone else validating it
Anyone can claim their own system works. While building this we found LangChain's AsyncPostgresSaver checkpointer failing against CockroachDB, and filed it upstream with a standalone repro and a proposed fix direction (langchain-ai/langgraph#8620).
A community contributor picked it up, built a complete fix along that direction, and verified it independently against both PostgreSQL 17.5 and CockroachDB v26.2.4. It's pending maintainer review as of this writing. Nobody on our team wrote that fix or ran that verification, which is a stronger signal than any benchmark we'd publish about ourselves.
Where to see it running
FleetMemory is our own entry in the CockroachDB x AWS "Build with Agentic Memory" hackathon, not a paid client project. It's public and MIT-licensed so anyone can read the gate logic or rerun the red team.
(opens the full-size image in a new tab)- Live demo, no login required: talk to agent Alex, come back as agent Maria, then hit "hallucinate a fact" and watch the gate catch it. https://pskmxv5sm4k3fj2rjoh2aoijka0ofgob.lambda-url.us-east-1.on.aws/
- Public repo (MIT license): the gate, the verifier, the red-team script, and the eval results. https://github.com/retailbox-automation/fleetmemory
- 1:15 demo video walking through the quarantine-and-reject moment. https://youtu.be/SDFpU-z9h2w
- The upstream issue: a fix verified by a third-party contributor, pending maintainer review. https://github.com/langchain-ai/langgraph/issues/8620
Want this built for your team? AI Integration Services — 400+ projects shipped since 2017.
Related services
Frequently asked questions
What is a provenance gate in AI agent memory?+
A provenance gate is a check that runs before any fact commits to an agent's memory. It asks whether the fact traces to a named source, like a direct customer statement, rather than trusting the agent's own confidence. A claim without a source quarantines instead of writing straight to memory.
Isn't a confidence score enough to filter out bad facts?+
No, and that's the exact gap our red team found. An agent can state a brand-new, unsourced fact with high confidence, and if nothing already in memory contradicts it, a confidence-only filter has nothing to compare it against. It sailed through our first gate 6 out of 6 times. The fix checks for a source, not a confidence number.
What happens to a fact the gate rejects?+
It doesn't disappear. It's written to a decision journal with the reason for the rejection, so anyone can query what was attempted, when, and why it didn't pass. Accepted facts that later get corrected aren't deleted either. The old value is marked invalid and kept, so the system can answer what it used to believe.
Is this the same as citing sources in a RAG answer?+
Related, but earlier in the pipeline. RAG citations tell a user where an answer's text came from. A provenance gate decides whether a claim is even allowed to become a durable fact in memory in the first place, before any agent ever retrieves and repeats it.
Can I run the red team myself?+
Yes. The repo is MIT-licensed and includes the red-team script and the eval results cited here. Point it at your own CockroachDB cluster and it reruns the same 30 attacks.