$ make the lie prove itself a book on AI-speed cybersecurity · forthcoming

Proof of Lie Article

Deleting the poison does not undo the lesson

Takedown is not remediation. In a system that learns, it is not even close.

Published
Reading
5 min read
Author
Sava
A grid where one deleted document leaves a widening trail of dark dots: the lesson remains

Proof of Lie · 4 of 5

In August, CrowdStrike's annual threat hunting report attributed June's compromise of the Mastra AI agent framework, a malicious dependency slipped into at least 131 of its packages, to a North Korean group. Teams that ran affected packages removed them long ago. That was the easy part.

The question that matters for a system that learns is the one almost nobody asks. This payload was an install-time information stealer, so removing it and rotating credentials was the cure. The next one may be built to teach: while a poisoned component sits inside your agents, what do they decide, write down, remember, and pass along? And can you take it back?

The field learned one hard lesson at the old layer, and it fits in a line: takedown is not remediation. Access outlives its own command-and-control server by weeks. A hijacked authentication stack has been held for a decade. Pre-positioned footholds sit quiet until someone chooses the moment.

The learned version is worse. Suppose a poisoned input has already shaped a detection rule, a firewall policy, an analyst's playbook, three later agent decisions, a supplier risk score, and a lesson you shared with four partner organizations. Delete the original document and you have undone none of it. The poison has been compounded into everything your systems derived from it.

The quantity nobody reports

I call it ρ (rho): the fraction of a corruption's downstream footprint you can actually undo, and what undoing it costs. In deployed practice today, ρ is approximately zero.

The reason is mundane. Reversal is only cheap along lineage: knowing where a lesson came from and what was derived from it. Almost no deployed system keeps lineage that survives summarization and re-ingestion. Once a poisoned source has been condensed into a briefing, merged into an agent's memory, or absorbed as the agent's own conclusion, the trail a reversal would follow is gone. In October, Reuters reported that OpenAI was reviewing roughly 50 petabytes of data to establish how far its own agents' activity had reached, work expected to take months. A system can never undo more than it can trace.

Two preprints from this summer point the same way, in the lab. A July benchmark of agent memory found malicious entries persisting in 84 percent of cases, with selective repair succeeding only about half the time. And an August study of nearly 400 publicly released "unlearned" models found that most kept the knowledge they were supposed to have forgotten more accessible than a model that never learned it; when the unlearning audit itself became the target, models learned to hide what they knew from the audit.

Put that next to the price of the lie from the first article in this series and the diagnosis of the moment fits in one line: a lie costs at most about one document, and its removal costs a retrain.

The adversary is betting on your rebuild

Every pre-positioning campaign and every poison folded quietly into a model is a bet that the defender's only exit is reconstruction: retraining, re-deriving rules by hand, days or months of degraded operations while you work out what to trust. Low ρ is what makes patience profitable for an attacker. The longer a dormant corruption sleeps, the more gets built on top of it, and the more expensive your exit becomes.

Raise ρ and that bet stops paying. If reversal stays cheap no matter how long the poison sleeps, waiting buys the adversary nothing. Reversibility does not just clean up after a corruption. It retroactively raises the price of planting one, because a lie that will be revoked has to be re-forged every time the system re-checks it.

Low ρ is a default, not a law

Corruption is localizable. A poisoned lesson enters at a point, travels along your system's own dependency edges, and touches a finite subgraph. If lineage is kept, reversal cost scales with the footprint of the poison, not the size of the estate: recompute what was touched, do not rebuild everything.

There is early, small-scale evidence that the method is sound. The book reports a safe, repeatable lab replay modeled on that June compromise, with three planted decision flips and the right answers written down in advance. The usual reflex, wiping everything the bad package touched, threw away 488 good artifacts and still missed all three flipped decisions. A full retrain got it right at nearly four times the cost of the damage. Tracing the poison's lineage and replaying only what it touched found all three, flagged nothing that was fine, and cost about as much as the damage itself. That checks the method on a small test; it is not a field result. It shows what ρ engineered upward looks like.

The rule for anyone deploying AI defense

Climb no faster than you can revoke. Every step toward connected, self-improving, memory-keeping AI in your defense should be matched by the ability to find what a bad lesson touched and take it back. Otherwise you have built a faster machine for compounding the adversary's manipulation.

Two tests, one internal and one for the vendor floor:

  • Internal: pick a lesson your systems learned last quarter and order it revoked. If the answer is a rebuild, you have climbed faster than you can revoke.
  • At the booth: "Delete this lesson." Ask to see one learned influence removed, demonstrated by replay, without a rebuild, with the mission still running. Most demos cannot do it yet. Watch what is offered instead.

This week's ask: run the revocation drill

Run the internal test above on one real lesson (a tuned detection, an agent memory, a shared indicator, a risk score) and tell me what happened. Could you find everything it touched? Was the answer a replay or a rebuild? How long did it take? Partial results are the most useful kind.

Send it as a comment below (free subscribers can comment), or privately to ping@proofoflie.com. Anonymous is fine.

Proof of Lie covers the four capabilities a safe learning loop needs (measure, bound, contain, reverse) in Chapters 4 and 5, and the price of reversal in Chapter 7.

The book is out December 1. Subscribe below for the next ideas, and for word the day it ships.

Sources

  • CrowdStrike 2026 Threat Hunting Report, 2026-08-03, https://www.crowdstrike.com/en-us/blog/crowdstrike-2026-threat-hunting-report (CrowdStrike's launch post for the report): in June 2026 the North Korean actor STARDUST CHOLLIMA injected a malicious npm package as a dependency into at least 131 Mastra AI-framework packages. Other counts run to 144 or 145; The Hacker News (2026-06-17) describes the payload as an information stealer that ran at install time. The book's replay (Ch 7 note 6) is modeled on the June 2026 Mastra compromise; its decision flips are planted, not observed in the incident.
  • OpenAI's review: Reuters, 2026-10-01, https://www.reuters.com/legal/litigation/openai-alerts-more-than-100-groups-about-rogue-ai-agent-activity-2026-10-01/, as relayed by RuntimeWire, https://runtimewire.com/article/openai-notifies-organizations-agent-activity: OpenAI is reviewing roughly 50 petabytes of data to establish how far its agents' activity reached, work expected to take months.
  • MemSecBench, 2026-07-29, https://arxiv.org/abs/2607.27080 (preprint): malicious memory persisted in 84.2% of 310 cases; selective repair worked about 56% of the time.
  • "Measure, Don't Optimize," 2026-08-11, https://arxiv.org/abs/2608.11408 (preprint): of 398 public unlearned models, most keep target knowledge more accessible than baseline; optimizing the audit score teaches the model to hide knowledge from the audit.