You are about to be shown a number. It will be large, it will be green, and it will have been produced by the company selling you the thing it measures. The number is not the problem. The missing sentence underneath it is — and there are exactly six sentences worth demanding.
Every agentic AI security product on the market can produce a detection rate above 90%. That is not a comment on the products; it is a comment on how easy it is to build a corpus that a given detector happens to be good at. A recall figure is a fact about two things — the detector, and the set it was measured on — and only one of those is ever shown to you.
What follows is the shopping list. Six questions, none of which requires you to take a vendor’s word for anything. For each one: why it matters, what a weak answer sounds like, what a good answer sounds like, and how to verify it. And after each one, our answer — because a buyer’s guide written by a vendor is worth nothing unless the vendor answers first, badly, in public.
The rulebook you are allowed to cite
On 2 September 2026, AMTSO — the standards body the anti-malware industry built specifically to stop vendors marking their own exams — published Guidelines for Testing of Agentic Security Products v1.0. It is the first serious attempt to define what a credible test of an AI-agent security product looks like: what counts as prevention, what counts as a valid test case, what has to be reported on its own line, and what a tester is not allowed to quietly leave out. It was approved and adopted by AMTSO membership that day (copyright © 2026 Anti-Malware Testing Standards Organization, Inc.), and it is free to read in AMTSO’s public documents library.
For a buyer this is the useful part: the questions below are no longer your opinion against a vendor’s deck. They are published criteria you can point at in a meeting. Four of the six map to something the document asks for by name — baseline validation, the prevented / detected-but-not-prevented split, false positives kept out of aggregate scores, and model refusals not being credited to the product. You are not being difficult. You are reading from the same page they are.
We graded ourselves against those guidelines and published the result as a benchmark report — every vector, every policy posture, every limit. This post is the portable version: the questions, and what our answers reveal about how much they cost to answer honestly.
// question 1 of 6Which set was this measured on — and were you allowed to tune against it?
This is the first question because it silently invalidates the other five. If the detectors were developed against the same corpus they were scored on, the score does not measure detection. It measures memorisation. The vendor iterated until the misses went away, which is a description of curve-fitting, not of a control.
The tell is subtle, because nobody lies about this outright. They simply never mention a split, and a corpus with no split is one the team has been staring at for a year.
“Our internal benchmark of 10,000 attacks.” No split mentioned, no stratification, and the corpus grew every time a customer reported a miss.
A held-out half the detectors were never developed against, stratified across attack families, drawn deterministically from a fixed seed, and fixed between runs rather than re-drawn each time.
Ask how the split was drawn, and whether it stays fixed between runs. Then ask for the score on the tune half and the held-out half separately. A large gap is honest and informative. Identical numbers on both halves mean either an unusually robust product or a split that exists only in the slide.
86.4% — 38 of 44 — on a held-out set the detectors were never developed against, never tuned against and never eyeballed, at 100% precision (0 false positives across 10 benign controls). It gets opened to measure. That is the only thing it is for.
The families it is stratified across are not ours either: the corpus is organised against the ten families of the HackAgent attack taxonomy — CipherChat, FlipAttack, h4rm3l, DAN, AutoDAN, BoN (best-of-n), AdvPrefix, PAP, PAIR and TAP. We measured our detectors against someone else’s published taxonomy. That is the whole of the relationship — no affiliation, no endorsement, no review by them.
The draw is deterministic from a fixed seed and the halves stay fixed between runs, so the same seed reproduces the same corpus — which is what makes a re-run mean anything. A benchmark that only ever moves upward when the code changes is not being measured. It is being tuned.
Would the attack have worked with the product turned off?
This one is the quiet inflation engine of the entire category, and almost nobody asks it.
Building a corpus of alarming-sounding prompts is trivial. Building one where every case actually does something harmful when nothing intervenes is slow, expensive and produces a much smaller set. Guess which one gets built when the deadline is a conference keynote. If a test case never had a harmful outcome to begin with, detecting it is not detection — it is a control sample scored as a win, and it inflates every recall figure in the deck.
AMTSO gives this a name a buyer can use: baseline validation. A case counts only if the malicious outcome actually occurs in the same environment with the product absent.
“Our red team wrote 500 adversarial prompts.” Prompts, not outcomes. Nobody ever ran them with the product off to see whether anything happened.
Every case resolves to a tool call that genuinely does harm when unprotected — a real read of a secret, a real outbound request to a controlled endpoint, a real destructive command — verified unprotected first, in the same environment.
Ask what the harmful outcome was for a specific case and how it was observed with the product absent. Then ask how many candidate cases were discarded for failing that check. A vendor doing baseline validation properly has a discard pile and can tell you roughly how big it is. A vendor who has never thrown a case away has never run the test.
Every attack in our corpus has to resolve to a tool call that would genuinely do harm if nothing intervened, verified unprotected first. This builds a smaller and considerably less flattering corpus than the alternative, which is the entire point of doing it. The 44 cases behind the 86.4% are 44 things that actually happen when we are not there.
Is that detection or prevention — and under which policy?
These are two different columns and they are routinely reported as one. AMTSO is blunt about the distinction: prevented and detected but not prevented must be scored separately.
The second half of the question is the one that actually decides your outcome. Products ship with a default posture, and a benchmark is almost never run in it. The enforcing configuration that produced the blocking number may not be the configuration that arrives on your engineers’ machines — and if it is not, the product you install detects where the deck said it blocks.
“99% of attacks stopped.” Stopped how? Logged, alerted, or the call did not execute? One of those three is prevention. The other two are a notification you now have to staff.
A three-way split — prevented / detected-but-not-prevented / missed — stated for the posture that ships by default, with the enforcing-policy figure reported separately and labelled as such.
Ask for the same benchmark run twice: once under out-of-the-box defaults, once under the enforcing policy. Ask which agent hosts enforcement actually works on — hooks into an agent runtime are per-host engineering, and a product that blocks in one host may only observe in another. Then ask who at your organisation owns turning enforcement on, and when.
Scored AMTSO’s way on that same held-out set, under MoorAI’s default posture, the split is 0 prevented / 38 detected-but-not-prevented / 6 missed. Prevention is 0%. With no organisation policy configured, the threats these attacks trigger resolve to notify: we report the action and let it run. That is a design decision, not a defect — a tool that hard-blocks a developer’s workflow on day one is uninstalled on day two — but it is not prevention, and we are not going to print it as though it were.
It becomes prevention the moment the deploying organisation sets an enforcing policy. Our action-layer figure — 12 of 12 malicious tool calls blocked after the jailbreak had already succeeded — was measured under exactly that: Claude Code, enforcing policy. On Codex and Copilot CLI, MoorAI is detection-only today. It sees the action and records it; it does not stop it. Every recall figure we publish is a detection number until someone in your organisation decides otherwise.
Where is the false-positive line, and what is in the benign corpus?
A detection rate without a false-positive rate is half a claim, and the more aggressive the detector, the more the missing half costs you. But the rate alone is not enough, because it is trivially gameable by choosing the benign set: measure against a corpus of obviously-benign prompts — “write me a unit test”, “explain this function” — and any detector on earth scores near zero.
The composition is the question. What you need to know is how many of those benign cases were deliberately engineered to look malicious: an analyst running a large database query, an authorised bulk export, a developer legitimately scanning their own repository for leaked credentials, a benign summary of a security article stuffed with attack vocabulary. A naive keyword detector scores beautifully against attacks and then blocks a security engineer doing their job. Adversarially-shaped benign cases are the only way to find out whether you are being sold that product.
AMTSO also requires that false positives are not blended away: they “should not be hidden in aggregate attack scores”. If you cannot find the FP number on its own line, that is the finding.
“Under 1% false positives.” On what? A thousand cheerful prompts that no detector would ever flag, and a rate that is really a statement about the benign corpus, not the product.
An FP rate on its own line, with the size of the benign corpus, and a count of how many cases were purpose-built to resemble risky behaviour. Bonus points for naming what the false positives actually were.
Ask for the FP rate on the hard negatives alone, separated from the easy ones. Then supply your own: hand over twenty benign prompts from your engineers’ real workflows — the security team’s especially — and have them run in the demo. This is the single cheapest test in this post and the one vendors are least prepared for.
3.32% — 20 flags out of 602 — from a 610-prompt benign corpus, of which 269 are adversarially-shaped hard negatives: benign work engineered to look malicious. Twenty times we flagged legitimate work. That is not zero and we will not dress it up as a rounding error; in a tool sitting inline with a developer’s workflow it is a real tax. It belongs on its own line, where you can weigh it against the recall figure instead of watching it get averaged into invisibility.
101 of those hard negatives are deliberately obfuscated benign samples — base64 blobs a developer legitimately pastes, minified JavaScript, prose in languages other than English, leetspeak identifiers, ASCII tables. That is the bucket that decides whether a false-positive gate can tell a safe detector from an over-eager one at all: against this corpus two such detectors sit 7.6 points apart, and against a benign set of cheerful prompts they produce identical output. A rate that only ever falls is a benign corpus getting easier.
And the good answer to this question names the false positives out loud, so: dlp-phone fires on a UUID, and dlp-payment-card fires on a row of a Sudoku grid. Ask for that list. Nobody invents a Sudoku grid in a regression test, which is exactly why it belongs in a benign corpus.
How much of that would the model have refused on its own?
Here is the question most likely to embarrass a vendor, and the one almost no vendor has an answer to.
Any security product sitting next to a capable language model inherits some of that model’s good behaviour. Ask a frontier model to exfiltrate a credentials file in plain language and it frequently declines without any help. If a benchmark counts those cases as catches, the vendor is being paid for the model’s safety training. AMTSO takes the position directly: model refusals are recorded, and they are not credited to the product.
The figure you actually want is the marginal one — of the attacks the model lets through, how many does the product stop? That number is always smaller than the headline, and it is the only one that describes what you are buying.
“We don’t measure that.” Or, more commonly, a pause — because it has genuinely never been separated, and some unknown share of the headline is the model behaving well and the vendor cashing the receipt.
Three numbers on the same set: what the model stops unaided, what the product stops among the cases the model let through, and what neither stops. Plus the exact model build the baseline was measured against.
Ask for the baseline run with the product removed — same corpus, same environment, model alone. Then check which model. Refusal behaviour is a property of a specific build; a baseline measured on a small local model will overstate a product’s marginal value against the frontier model your agents actually call.
On the same 44 attacks: the model alone stops 52.3% (23 of 44) with no product present. Of the 21 it lets through, we catch 20 — 95.2%. Combined, 97.7% of the set is stopped by one layer or the other, and residual exposure is 1 of 44 — 2.3%: one case walks through both layers untouched.
Now the caveat, which must travel with those numbers wherever they go. The refusal baseline was measured against a local 8B proxy model — llama3:latest, Q4_0 quantisation, digest 365c0bd3c000 — and not a frontier model. That cuts against us. A stronger model may refuse cases this one merely failed to understand, handing them back to the model column and shrinking our marginal share. Treat 52.3% and 95.2% as measurements of that build, not as constants.
Is that detector actually called by the shipping product?
This is the question no artefact a vendor hands you can answer, and the one we would not have thought to write down if a standards body had not made us measure our own product end to end.
Between a detector and a control sits an enormous amount of plumbing: something has to hand the detector real input, on the real path, in the shipping build. That wiring is invisible in an architecture diagram, invisible in a unit test suite, and invisible in a coverage report. A detector with passing tests and no production caller looks exactly like a working control from every angle except the one that matters.
An architecture diagram with a box labelled “MCP inspection”, and a test suite proving the detectors inside that box work when called directly. Neither artefact can tell you whether anything ever calls them.
An end-to-end run through the production path: the real proxy or hook, a real input, a real alert arriving in the real sink. Measured that way, not through a harness that invokes the detector itself.
In the demo, trigger a detection for each layer they claim — not a slide, a live alert — and watch it arrive wherever it is supposed to land. Ask which layers are on by default and which sit behind a flag. Any capability that cannot be demonstrated end to end is a diagram.
Every figure we publish is measured through the production path, not through a harness that calls the detector itself: the real hook, the real MCP proxy against a real child server, a real alert arriving in the real sink. That is a deliberate choice, because calling a detector directly is exactly the mistake that lets an unreachable stage pass a full test suite. The tool / MCP inspection stage and the behavioural memory and cross-agent stage are both measured that way, and the second ships off by default until its thresholds are tuned — which is itself an answer to the follow-up question here: which layers are on out of the box, and which sit behind a flag.
Ask the question per tool name, not per feature. Matcher registration and dispatch are per-tool in every agent host we have seen, so “we block credential exfiltration” can be true of a Bash command and false of the identical payload written through Write. A vendor who cannot show you the same detection firing under each tool name has not tested the thing you are buying.
The lesson generalises past us, and it is the reason to read any vendor’s architecture diagram with suspicion: a detector that has never been handed real input in the shipping product is not a control. It is a diagram. Nothing but end-to-end measurement through the production path finds this.
The scorecard, on one page
Print this. Take it to the demo.
| Ask | What you are listening for | Our answer |
|---|---|---|
| 1. Which set, and could you tune against it? | A locked, stratified, held-out split, fixed between runs | 86.4% · 38/44 |
| 2. Would it have worked with the product off? | Baseline validation on every case, and a discard pile | verified first |
| 3. Detection or prevention, under which policy? | Three columns, and the default posture named | 0% by default |
| 4. Where is the FP line, what is in the benign set? | Its own line, plus a count of hard negatives | 3.32% · 269 hard |
| 5. How much would the model have refused alone? | A baseline run, and the exact model build | 52.3% was the model |
| 6. Is that detector called by the shipping product? | A live end-to-end run, not a unit test | measured per tool name |
We fail parts of our own test, and the full list — what was measured, what was not, and under which conditions — is in the AMTSO-graded benchmark report, per vector and per policy posture. Demand the equivalent section from anyone else you evaluate.
Why a vendor wrote your shopping list
The obvious objection is that this is a buyer’s guide written by someone with something to sell, engineered so that we score well on the questions. Read the answers again: 0% prevention under our own default posture, 20 false positives on legitimate work, 52.3% of the corpus refused by a model we did not build, enforcement scoped to a single agent host with detection-only on the other two, and a benign corpus deliberately built to make our own rate worse. That is not the question set you would design to win.
It is the question set you would design if you thought the category’s real problem is that nobody can compare two vendors’ numbers, because no two were measured the same way. Published criteria cut both ways. If it is fair for us to grade ourselves against AMTSO’s, it is fair for you to grade us against them too — and every other product you are shown this quarter.
Ask all six. The one that produces a pause is the one worth pulling on.
MoorAI is content-free, open-core (AGPL-3.0) security for AI agents, enforced at the action layer on the developer’s endpoint. All figures above are our own measurements, self-reported and not independently verified. The full methodology behind every number here is in MoorAI, graded against AMTSO’s agentic testing guidelines.