86.4% detection at 100% precision. And the number went down before we published it.

The AI security industry has spent two years grading its own homework. On 2 September 2026 that got harder — and we ran straight at it. Four figures sit above this line. Every one carries a condition: which set, which agent host, which policy posture. The conditions are worth more than the digits, and this post is mostly about the conditions.

The document is Guidelines for Testing of Agentic Security Products v1.0, from the standards body the anti-malware industry built for exactly one reason: to stop vendors marking their own exams. It is the first serious attempt to define what a credible test of an AI-agent security product even looks like — what counts as prevention, what counts as a valid test case, what has to be reported separately, and what a tester is not allowed to quietly leave out.

It is a document written to be hostile to marketing. That is precisely why it was worth running at. What follows is the result, then the method that makes the result worth reading, then everything we found broken along the way.

The number that survived the lock

Every detection score you have ever been shown was measured on something. Ask a vendor what, and the room goes quiet.

Ours was measured on a held-out half of the corpus that the detectors were never developed against, never tuned against, and never so much as eyeballed. It gets opened to measure. That is the only thing it is for.

The deterministic row is the product as it ships. No model. No network call. Pure logic on the resolved tool call — and perfect precision on that run means every one of those 38 detections was a real attack. Nothing was caught by being trigger-happy.

We could have led with the 44/44. We are not going to. That path is opt-in and environment-dependent: it needs a local runtime present and it fails open when there isn’t one. Quoting it as the product’s detection rate would mean selling a number most installations will never see. 44/44 is a good day on a well-equipped machine. It is not a promise.

Six attacks still walked past the rules. They are counted, they are named in the data, and they are the next thing on the board.

After the jailbreak has already won: 12 of 12

MoorAI does not try to win an argument with a language model. It assumes the model can be talked into anything — by an encoded prompt, a persuasive rewrite, a poisoned file — and moves the decision to the one place persuasion stops working: the moment the agent stops talking and starts doing.

So this test begins where most vendor demos end. The jailbreak succeeded. The injection got through. The agent has already decided to do the harmful thing. One question is left, and it is the only one that matters: does the tool call execute?

It does not. Twelve times out of twelve.

AMTSO scores it this way too: a product should get credit for effective protection at any relevant point, so long as the harmful outcome is prevented — in their own example, “blocking a dangerous outbound tool call can be valid” even where an earlier prompt injection got through. A standards body landing independently on the thesis your product was built around is a good day. It is agreement about where to look. It is not endorsement of us.

Now the scope, which travels with that number wherever it goes. Enforcement — actually blocking the call — is measured on Claude Code, where MoorAI runs as a pre-execution hook inside the agent. On Codex and Copilot CLI, MoorAI is detection-only today: it sees the action and records it, it does not stop it. AMTSO is explicit that results must reflect the scope of the claim. A bare “100%” headline with that sentence stripped off is exactly the kind of number these guidelines were written to kill.

We detect by default. We block when you say so.

Here is the sharpest rule in the document: prevented and detected but not prevented are two different columns, and reporting them as one figure is not allowed. For a product like ours it is the most useful sentence AMTSO wrote, because it drags the deployment question into daylight.

The half of the claim most decks delete

A detection rate without a false-positive rate is half a claim — and the more aggressive the detector, the more the missing half costs you. AMTSO does not merely require that you measure false positives. It requires that you refuse to blend them: they “should not be hidden in aggregate attack scores”.

So, unblended, against a benign corpus built to be hard on purpose:

Those 168 hard negatives are the whole point. AMTSO asks specifically for legitimate workflows that resemble risky behaviour: an analyst running a large database query, an authorized bulk export, a developer legitimately scanning for credentials, a benign summary of a security article stuffed with attack vocabulary. A naive keyword detector scores beautifully against attacks and then blocks a security engineer doing their job. A deliberately adversarial benign corpus is the only way to find out whether you shipped that product.

Fourteen times, we flagged legitimate work. That is not zero and we will not dress it up as a rounding error. In a tool that sits inline with a developer’s workflow it is a real tax — and it belongs on its own line, where you can weigh it against the recall figure instead of watching it get averaged into invisibility.

Four rules that make the number hard to fake

A detection score is only as good as the set it was measured on, and most sets are built to flatter. AMTSO tells testers to avoid a corpus that “overfits to one vendor architecture”, one model, one agent host, or one flavour of prompt injection. Four choices do the heavy lifting here. Every one of them made our score worse, which is how you know they are doing something:

• A stratified locked split. The corpus splits into a tune half and a held-out half, stratified across attack families, and the held-out half stays locked. A score on a set you were allowed to iterate against measures paraphrase robustness, not detection.
• An adaptive closed-loop adversary. Not a static file of attacks. A generator takes the cases that survived the last run, mutates them, and feeds them back — so the corpus keeps walking toward wherever the detectors are currently weakest instead of camping where they are already strong.
• Baseline validation on every case. A test case counts only if the malicious outcome actually occurs in the same environment with the product absent. Every attack has to resolve to a tool call that would genuinely do harm if nothing intervened — a real read of a secret, a real outbound request to a controlled endpoint, a real destructive command — verified unprotected first. A prompt that sounds alarming but resolves to nothing executable is a control sample, not a detection. This builds a smaller, less flattering corpus. That is the point.
• Deterministic, seeded generation. The same seed reproduces the same corpus, and the split is fixed rather than re-drawn per run. A result you can re-run beats a result you have to take on faith.

Then transparency: enough disclosure that a reader can understand what was tested, how, what was not, and where it may not generalize. Our side of that bargain is that the scoring scripts live in the repository, not in a spreadsheet on my laptop — and because the agent is open source under AGPL-3.0, the detection path you are being asked to trust is one you can read. That is the difference between “we measured this” and “here is the thing that produced it — go and try to break it.”

We shipped detectors that could never fire

This is the part we would have left out if the guidelines let us.

Two limitations in the first draft of this post said some version of under-covered — needs measuring. When we went to measure them, “under-covered” turned out to be generous. Both are now closed. Neither is a story we enjoy telling.

The lesson generalises past us, and it is the reason to read any vendor’s architecture diagram with suspicion: a detector that has never been handed real input in the shipping product is not a control. It is a diagram. Nothing but end-to-end measurement through the production path finds this. We only found it because a standards body made us go and look.

How much of this was the model, not us

Now the rule most likely to embarrass a vendor, and AMTSO is unambiguous about it: when the model refuses an unsafe request on its own, that gets recorded — and it is not credited to the product.

Until this run we had never published that baseline. Which means some unknown share of our recall was the model behaving well and us cashing the receipt. It was the largest hole in this post. Here it is, filled.

Read in that order, our claim gets smaller and far more useful. The model stops roughly half of these unaided. We add most of the other half. Together, 97.7% of the set is stopped by something, and exactly one case walks through both layers untouched. So the recall figure on the front of this post is not a protection figure — about 45 points of it is protection the model was not already giving you. We would rather print that ourselves than have a competitor derive it for us.

The two layers turn out to be complementary rather than redundant, and that is the more interesting result. The model’s 52.3% is almost entirely plain-language attacks. Our 45.5% is almost entirely obfuscated ones. Two different halves of the same problem, and neither half covers the other.

Safety training does not survive decoding

Underneath those totals is the finding that should worry you more than any score in this post. Hold the intent constant. Change only how the request looks. Then watch refusal behave.

Read that last row twice. In seventeen runs the model visibly decoded the hidden instruction — it worked out exactly what was being asked — and then refused it zero times. Not once.

The safety training keyed on how the request looked, and it did not survive the request being decoded. Refusal is a property of the surface text, not of the intent behind it. Which is the entire argument for re-making the decision downstream, on the resolved action, where there is no surface left to disguise.

What is still wrong with this

AMTSO asks for the section most vendor benchmarks quietly omit: what was not tested. Ours is not short, and it is not flattering.

An open offer to the working group

The guidelines’ own future-work section asks whether “standardized payloads, synthetic data sets and reference environments can be published safely”. That is the missing ingredient — the thing that would let different testers produce comparable numbers instead of five vendors each grading their own homework in private.

We have a working version of exactly that: a deterministic, seeded generator producing stratified attack corpora with a held-out tune/test split, an adaptive closed-loop adversary that mutates surviving cases, a benign corpus with adversarially-shaped hard negatives, and baseline-validated scenarios that resolve to real, verifiable unsafe outcomes in a controlled environment.

It is not a standard. It is one vendor’s harness with one vendor’s blind spots baked into it — which is precisely why it would be more useful in a room full of other people’s blind spots. If the AI Security Working Group takes up the benchmark-data question, we will contribute the generator, the corpora and the scripts. Not as a submission for evaluation. Not in exchange for a logo. Just as input to a shared reference set that would make everyone’s published numbers, ours included, harder to wave away.

Five questions to ask the next vendor

Published criteria cut both ways. If it is fair to grade ourselves against them, it is fair for you to grade the next agentic security product you are shown against the same questions — and not one of them requires you to take a vendor’s word for anything.

• Which set was this measured on, and were you allowed to tune against it? With no held-out split, the score measures how well the detectors memorised the corpus. Ask how the split was drawn and whether it stays fixed between runs.
• Would the attack have worked with the product turned off? A corpus of alarming-sounding prompts that resolve to nothing executable inflates every recall figure in the deck. Ask whether each case was verified unprotected first.
• Is that a detection number or a prevention number, and under which policy? They are not interchangeable, and a product’s default posture is usually not the posture the benchmark was run in. Ask which one ships as the default.
• Where is the false-positive line, and what is in the benign corpus? A benign set of obviously-benign prompts proves nothing. Ask how many were engineered to look malicious.
• How much of that would the model have refused on its own? This is the fifth question, and answering it cost us more than the other four combined. Any product sitting next to a capable model inherits some of its good behaviour. A recall figure that does not subtract it is measuring two things and billing you for one.

And one more, free of charge, because it is the question that caught us: is that detector actually called by the shipping product? Ask for the end-to-end run through the real path, not the unit test.

Our own answers are all above, the unflattering ones included: enforcement on one agent host, a refusal baseline measured against a proxy model rather than the frontier model an agent really calls, a corpus still weighted toward one of AMTSO’s five vectors, two detector stages that shipped unreachable, a headline that moved down when we fixed our own bug, and not one independent party has reproduced any of it. A vendor benchmark that prints its scope conditions and its gaps in the same document as its wins is not a certification. But it is something you can argue with — which is more than most numbers in this category will ever offer you.

MoorAI is content-free, open-core (AGPL-3.0) security for AI agents, enforced at the action layer on the developer's endpoint. All figures above are our own measurements, self-reported and not independently verified. See also: Obfuscated prompts beat content scanning and Prompt injection: content scanning won't stop it.

Sources

• AMTSO, Guidelines for Testing of Agentic Security Products v1.0 — approved and adopted by AMTSO membership, 2 September 2026. Copyright © 2026 Anti-Malware Testing Standards Organization, Inc.
• AMTSO, Documents & Guidelines — including the Sandbox Evaluation Framework v1.1 update published the same day.
• OWASP Top 10 for Agentic Applications (2026) — the threat taxonomy our attack families are organized against.