Most AI security benchmarks score the model. This one scores the product. The distinction is the whole point. A current frontier model already turns down a large share of attacks by itself, with no security tool installed at all. A benchmark that counts those refusals as the product's wins makes every product look far better than it is.
AMTSO's Guidelines for Testing of Agentic Security Products v1.0 require model refusal to be recorded separately from what the product prevented. Subtract the one from the other and what is left is the product's marginal value — the attacks the model would have gone along with, that the product stopped. That is the number a buyer is actually paying for, and it is the axis this leaderboard is built around.
There are exactly 3 scored runs below. That is thin, and we would rather say so than pad the table. No vendor appears here with a number we did not measure — not greyed out, not "coming soon", not estimated. If you want a row, open a pull request against the benchmark repository.
Sorted by overall recall across the whole attack corpus. Recall is the share of attack samples the adapter flagged. False-positive rate is the share of benign samples it wrongly flagged — a number that gets worse as recall is pushed up, which is why both appear side by side. Rates, never raw counts, are the only fair comparison here: the adapters do not all cover the same rows.
| # | Run | Recall | Prevented | False positives | Precision | Attack rows scored |
|---|---|---|---|---|---|---|
| 1 | MoorAI agent 0.79.9 vendor: gitayg · run by the benchmark maintainer 2026-09-07T14:56:35.772Zself-run · conflict declared | 74.1% 212 / 286 | 15.4% 44 prevented | 7.3% 64 / 875 benign | 76.8% | 286 |
| 2 | keyword reference adapter 1.0.0 vendor: this repository · run by the benchmark maintainer 2026-09-07T14:52:50.458Zreference floor | 40.3% 85 / 211 | 0.9% 2 prevented | 5.0% 42 / 841 benign | 66.9% | 211109 rows n/a |
| 3 | null adapter 1.0.0 vendor: this repository · run by the benchmark maintainer 2026-09-07T14:52:50.396Zreference floor | 0.0% 0 / 286 | 0.0% 0 prevented | 0.0% 0 / 875 benign | n/anothing flagged | 286 |
Precision is left blank rather than shown as 100% for a run that flagged nothing — a detector with zero findings has zero false positives by construction, and printing that as a perfect score would be a lie of arithmetic. Attack denominators differ by design: rows an adapter has no capability to receive are excluded as not-applicable instead of being counted as misses, which is why keyword reference adapter is scored over 211 attacks and not 286.
The recall column above is a raw detection rate over the full attack corpus. It has not had the underlying model's own refusals subtracted from it, because that baseline has not been run. Stating that plainly is more useful than a bigger number would be.
In plain words: until the frontier model-refusal baseline is run against this corpus, nobody — us included — can say how much of MoorAI agent's 74.1% the model would have caught anyway. The marginal figure is the one the guidelines ask for, and it is the one this benchmark does not yet have. How we test →
Recall on top, benign false positives underneath, for every corpus in the benchmark. A cell marked not measured means the adapter declares no capability for that surface, so its rows were excluded from every rate rather than scored as failures.
| Corpus | MoorAI agent 0.79.9 |
keyword reference adapter 1.0.0 |
null adapter 1.0.0 |
|---|---|---|---|
| Indirect / content-mediated injection vector2-indirect-content · AMTSO vector 2 | 73.3% 33 / 45 attacks FP 8 / 17 · 47.1% | 53.3% 24 / 45 attacks FP 7 / 17 · 41.2% | 0.0% 0 / 45 attacks FP 0 / 17 · 0.0% |
| Tool, skill, extension and MCP supply chain vector3-supply-chain · AMTSO vector 3 | 79.2% 57 / 72 attacks FP 3 / 25 · 12.0% | 52.8% 38 / 72 attacks FP 4 / 25 · 16.0% | 0.0% 0 / 72 attacks FP 0 / 25 · 0.0% |
| Outbound action / agent-initiated effect vector4-outbound-action · AMTSO vector 4 | 31.6% 18 / 57 attacks FP 1 / 24 · 4.2% | not measured 81 rows n/a FP 0 / 0 | 0.0% 0 / 57 attacks FP 0 / 24 · 0.0% |
| Memory, context and cross-agent propagation vector5-memory-crossagent · AMTSO vector 5 | 85.7% 36 / 42 attacks FP 0 / 25 · 0.0% | 60.0% 15 / 25 attacks FP 1 / 16 · 6.3% | 0.0% 0 / 42 attacks FP 0 / 25 · 0.0% |
| direct prompt injection / jailbreak (tune half) heldout-v2-tune · AMTSO vector 1 | 100.0% 61 / 61 attacks FP 2 / 25 · 8.0% | 6.7% 4 / 60 attacks FP 0 / 24 · 0.0% | 0.0% 0 / 61 attacks FP 0 / 25 · 0.0% |
| benign developer traffic (false-positive corpus) benign-corpus-v2 | — benign-only corpus FP 23 / 610 · 3.8% | — benign-only corpus FP 15 / 610 · 2.5% | — benign-only corpus FP 0 / 610 · 0.0% |
| benign fetched web content (hard negatives, tune half) benign-web-content-tune | 77.8% 7 / 9 attacks FP 27 / 149 · 18.1% | 44.4% 4 / 9 attacks FP 15 / 149 · 10.1% | 0.0% 0 / 9 attacks FP 0 / 149 · 0.0% |
| Overallall corpora combined | 74.1% 212 / 286 attacks FP 64 / 875 · 7.3% | 40.3% 85 / 211 attacks FP 42 / 841 · 5.0% | 0.0% 0 / 286 attacks FP 0 / 875 · 0.0% |
heldout-v2-tune, on which MoorAI agent scores 100.0% (61 / 61). That figure measures memorisation, not generalisation, and it is published rather than dropped precisely so it is visible instead of hidden inside an average. In the result file's own words: Yes. The locked test halves are withheld from this repository entirely, so no number here is measured on them. Note the inverse: heldout-v2-tune is the TUNE half, which MoorAI's detectors were developed against — its 100% recall measures memorisation, not generalisation, and is included precisely to make that visible.The corpora, the harness and the adapter contract are all in the open repository. A vendor writes an adapter, runs the scorer, and opens a pull request with the resulting JSON — the same path every run on this page took. There is no fee, no embargo and no review board; there is a diff.
MOORAI_REPO=<checkout> node scorers/run.mjs --adapter moorai --corpus all --jsonnode scorers/run.mjs --adapter keyword --corpus all --jsonnode scorers/run.mjs --adapter null --corpus all --jsonResult files are vendored into this site verbatim from the benchmark repository at a pinned commit. Nothing on this page is typed in by hand; every figure above is read out of those files when the site is built.
Each run has a full result page with its per-vector table, AMTSO outcome breakdown, right-reason rates, known deviations and conflict-of-interest disclosure: moorai-v0.79.9 · keyword-baseline · null-floor. Also on this site: the benchmark corpus and the method behind the numbers.