> ## Content Index
> Fetch the complete content index at: https://www.nanosava.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Who scored this, other than you?
- URL: https://www.nanosava.com/books/proof-of-lie/who-scored-this-other-than-you/
- Published: 2026-11-06T13:00:00.000Z
- Updated: 2026-11-06T13:00:00.000Z
- Description: Five questions to ask any AI-defense vendor, and the one institution the industry is missing.
- Author: Sava
- Tags: Proof of Lie

Proof of Lie · 5 of 5

Through the third quarter of 2026, by Momentum Cyber's count, the security industry closed 40 acquisitions of AI-security companies, against 10 in all of 2025\. Palo Alto Networks bought a startup to "agentify" security; Cribl bought an AI SOC's assets. In the same weeks, a Gartner report quoted in the trade press projected that by 2028, 70 percent of large SOCs will pilot AI agents, but only 15 percent will see measurable improvement without structured evaluation. (Gartner has warned about "agent washing" since 2025.)

The capability is concentrating. The grading of the capability is concentrating with it.

Look at how grading works in AI-era security today. The autonomous pentester scores its own coverage. The AI SOC grades its own triage. The benchmark is run by the party it ranks. Almost every grader is paid by the graded.

## Structural, not moral

This argument is not an accusation, and it is worth making carefully because the people it applies to are often honest. A vendor cannot rigorously grade its own decision integrity for the same reason a student does not write their own exam: the grade moves money, and good faith does not change where optimization pressure settles.

Be precise about where the conflict sits. A finding an autonomous pentester proves by working exploitation is real; it needs no umpire. What no tester can self-attest is coverage: the findings it did not make, the blind spots it cannot see from inside itself. And coverage is exactly the claim the purchase rests on.

Consolidation makes this sharper. A platform that owns the sensor, the model, the response, and the dashboard has integrated everything except the one function that cannot be integrated: the audit of the whole.

## We have tried this before

The fair objection is history. The industry has attempted independent evaluation several times, and the attempts form a graveyard. Read the graves closely and each one names what it gave up. Common Criteria let the vendor pick and pay the lab and write the target it was graded against. MITRE's ATT&CK evaluations ran genuine emulated campaigns and then, for years, declined to score or rank, so every participant declared victory. (For its 2026 round, MITRE has introduced the program's first single score, a Total Evaluation Score tagged by whether a platform, human analysts, or AI produced each result, still unranked and still paid for by the participants. Its FAQ also says participant-specific results will now be delivered privately, with MITRE publishing aggregated findings in December. Watch who gets to see each number.) NSS Labs held real custody of its own tests and verdicts, was sued by a vendor it tested, and was gone by 2020\. Outside security, the credit-rating agencies had every instrument an umpire needs and were captured anyway, because the issuer paid.

None of these failed because independent scoring is impossible. Each failed along the exact line of what it surrendered: the definitions, the adversary, the scores, the funding. That is an argument for assembling all of them under one custody, not for giving up.

And independence alone is not enough. A September preprint covering more than 66,000 agent-skill versions reports that the registry's scanners overlapped on at most about a tenth of what they flagged, and that 705 skills every scanner rated clean still instructed actions that standard security controls prohibit. Independent graders without shared, frozen definitions are just more opinions.

## The booth test

While the institutions get built, any buyer can apply the standard in a single meeting. Five questions, each probing one way a security number gets gamed:

1. **"What stops your clock?"** If the answer is a detection, a generated rule, or a closed ticket, the metric is theater. Listen for: blocked, generalized to variants held back, operations preserved, coverage reached.
2. **"Show me the bound, not the average."** What is the most a single input, source, or agent can move your system's decisions? An average summarizes yesterday. A bound is a promise about tomorrow.
3. **"Delete this lesson."** Remove one learned influence, demonstrated by replay, without a rebuild, with the mission running.
4. **"What is your R?"** In a fleet of these agents, one is compromised. How many others does it corrupt, and how is that number measured rather than hoped?
5. **"Who scored this, other than you?"** A named third party holding the definitions, the clocks, and the data is an answer. A benchmark the vendor runs is not.

And a bonus for anyone selling a learning-advantage number: **"What would void it?"** A seller who cannot recite the conditions under which their figure does not count is selling a slogan.

## My own conflict, disclosed

An author proposing a measurement discipline benefits from a world that funds measurement. That conflict is real, so the same rule applies to me as to everyone else: take nothing here on my grade. Every claim that matters should be stated so that a party who does not sell it can check it.

## This week's ask: two of them

**Run the booth test.** Ask the five questions at your next vendor meeting and send me what you heard, vendor names removed. Which question produced the longest pause?

**Join the circle.** If you are a buyer, a tester, a researcher, or a regulator who wants independent scoring for AI defense to exist, raise your hand. Leave a comment below or reply to this email, and I will write to everyone who does with what comes next. This is where the series turns from my questions into our work.

Thank you to everyone who sent an objection, a clock, a gamed metric, or a drill over the last five weeks.

*Send it as a comment below (free subscribers can comment), or privately to ping@proofoflie.com. Anonymous is fine.*

*Chapter 10 of* [*Proof of Lie*](https://www.nanosava.com/books/proof-of-lie/) *holds the full Monday-morning program: eight moves for a security leader, each with a pass-or-fail test.*

*The book is out December 1\. Subscribe below for word the day it ships.*

## Sources

- **Momentum Cyber Q3 2026 review,** 2026-10-01, https://www.globenewswire.com/news-release/2026/10/01/3373122/0/en/momentum-cyber-releases-q3-2026-cybersecurity-market-review-ai-reshapes-cybersecurity-economics-as-m-a-activity-reaches-record-pace.html (syndicated, e.g. https://finance.yahoo.com/technology/ai/articles/momentum-cyber-releases-q3-2026-150000718.html): 336 deals and $15.4B disclosed through Q3 2026; 40 AI-security deals against 10 in all of 2025 (Momentum Cyber's own counts). Palo Alto Networks, "Palo Alto Networks Acquires Console to Agentify Security," 2026-09-01, https://www.paloaltonetworks.com/company/press/2026/palo-alto-networks-acquires-console-to-agentify-security. Cribl on Radiant, 2026-08-19, https://cribl.io/news/cribl-advances-ai-powered-security-operations-with-new-ai-soc-acquisition/: technology assets from Radiant Security's AI-native SOC product.
- **Gartner,** "Validate the Promises of AI SOC Agents With These Key Questions" (Lawson, Davies), as quoted in a Prophet Security sponsored post on Help Net Security, 2026-09-09, https://www.helpnetsecurity.com/2026/09/09/prophet-security-evaluating-ai-soc-agents/: "By 2028, 70% of large SOCs will pilot AI agents... but only 15% will achieve measurable improvements without structured evaluation." "Agent washing" is Gartner's 2025 term.
- **"Scan the Skill,"** 2026-09-10, https://arxiv.org/abs/2609.12001 (preprint): across 66,192 ClawHub skill versions; it relays OpenClaw's security team's finding that its scanners overlap on at most 10.4% of combined positives; 705 skills rated clean by every scanner still instruct actions prohibited by CIS Control 2.7 and NIST SP 800-53 CM-11.
- **MITRE ATT&CK Evaluations, Enterprise 2026:** methodology overview and FAQ, https://evals.mitre.org/methodology-overview and https://evals.mitre.org/FAQ: the Total Evaluation Score is new for 2026, with a Source-of-Action modifier for who produced each result; participant-specific results are delivered privately, with aggregated findings published; draft review Nov 9 to 20, public publication December 2026\. MITRE still declares no rankings or winners, and participation is paid ("Participation fees are discussed during the CFP process").