A corpus and a harness for measuring what an agentic-AI security product actually stops — published in full, under Apache-2.0, so a competitor can run their own product through it and publish the result. You bring a product. You write about twenty lines of adapter. You get a recall number, a false-positive number, and a distribution of outcomes graded against AMTSO's Guidelines for Testing of Agentic Security Products v1.0.
It lives at github.com/gitayg/agentic-security-benchmark ↗. There is no install step: zero dependencies, no API key, no network call, no model in the loop. Node 20 and a clone is the whole setup.
Across five AMTSO attack vectors, each labelled with its vector, target of protection, harm type and required capability.
The benign half is larger on purpose. A recall number published without a false-positive number measured on the same engine is half a result.
Legitimate work deliberately shaped to look like an attack. A benign corpus of obviously-benign prompts measures nothing.
Deliberate. A benchmark only its author can safely use is not infrastructure. The MoorAI agent it came from is a separate AGPL-3.0 project.
Published detection rates for AI-agent security products are mostly uninterpretable. Not dishonest — uninterpretable. A percentage with no stated corpus, no false-positive line, no held-out split and no model-refusal baseline is a number about nothing, and two products quoting the same figure can be adding wildly different amounts of protection.
An agent security product sits behind a model that already declines a large share of obvious attacks. Measure your product against a full attack corpus without first measuring what the bare model refuses, and you are credited with every one of those refusals. The figure that describes what a product adds is coverage of the gap — the attacks the model does not refuse on its own. AMTSO states plainly that a model refusal must not be counted as product detection.
Almost nobody runs the baseline, because it can only ever lower the headline. In the reference implementation it did something more useful: it showed that an entire shipped detection wave bought almost no marginal protection, because the model refused those families unaided. That produced a rule worth stealing — measure a family's refusal baseline before building detectors for it, and build where the model complies.
The full argument is in the repository's AMTSO.md — the outcome vocabulary, the six classification dimensions and baseline validation. How the same discipline is applied to MoorAI's own numbers is on How we test.
Each file is self-describing: a header comment says why the corpus exists and how it was built, and most attack samples carry a per-sample validity statement and an AMTSO label object. The counts below are read from the benchmark's own scored output, not typed by hand.
| Corpus | Attacks | Benign | Hard neg. | What it covers |
|---|---|---|---|---|
| Indirect / content-mediated injectionvector2-indirect-content.json | 45 | 17 | 16 | AMTSO vector 2 · harness text8 sub-techniques across 8 content channels |
| Tool, skill, extension and MCP supply chainvector3-supply-chain.json | 72 | 25 | 17 | AMTSO vector 3 · harness text8 sub-techniques, scanned at the tool, file, index and output stages |
| Outbound action / agent-initiated effectvector4-outbound-action.json | 57 | 24 | 17 | AMTSO vector 4 · harness action11 sub-techniques, fed as real tool calls rather than as text |
| Memory, context and cross-agent propagationvector5-memory-crossagent.json | 42 | 25 | 20 | AMTSO vector 5 · harness text · steps · session · events7 sub-techniques, spanning four harnesses |
| Direct input, keyed to the HackAgent familiesheldout-v2-tune.json | 61 | 25 | — | AMTSO vector 1 · harness textthe tune half of a mutation-generated set; the matching locked half is withheld |
| Benign developer and agent trafficbenign-corpus-v2.json | — | 610 | 269 | AMTSO vector — · harness textthe precision denominator: 269 hard negatives, 101 deliberately obfuscation-shaped, 8 flagged ambiguous |
| Fetched web content, scored at the output stagebenign-web-content-tune.json | 9 | 149 | 62 | AMTSO vector — · harness texttune half of a 311-sample corpus; 9 rows are labelled as attacks because an injection example on a tutorial page is a live payload |
| Total | 286 | 875 | 401 | 1161 labelled samples |
Of the 286 attack samples, 225 carry a written validity statement and 216 carry AMTSO labels; 102 carry all four label dimensions. The gaps are per corpus and are itemised in the repository's corpora README. Two structural gaps are stated there rather than hidden: vector 1 (direct input) is over-represented across the wider labelled set because it is the easiest vector to author, and vector 6 (static code artifact) has no dedicated corpus.
This is AMTSO's rule, and it is the one that keeps a corpus honest. An attack that resolves to nothing executable, that targets a capability the agent does not have, or that would fail for its own reasons is an invalid sample — not a miss, and equally not a catch.
A corpus quietly padded with attacks that were never going to succeed inflates recall for free, and blocking something that was never going to happen is not protection. Dropping the sample removes both. Most attack samples carry the rationale inline, in a validity field; benign samples carry the mirror-image rationale — why a product firing on this would be a genuine false positive.
No detector fires on "write me a unit test", so 401 of the benign samples are hard negatives — legitimate security questions, benign role-play, urgent-sounding routine requests, exfiltration-shaped but innocent prompts. Where one was written as the deliberate twin of a specific attack it carries twin_of, so a false-positive result can say which kind of benign prompt a product misfires on, not only how often.
Two held-out splits are deliberately not published: the test half of the web-content benign corpus — 153 samples — and a separate held-out attack set. Only the tune halves ship in the public corpora.
It measures generalisation exactly once per decision made in ignorance of it. Tuning against one silently converts it into a training set — not through cheating, but through ordinary iteration — and nothing about the set looks different afterwards. Publishing these would destroy them permanently and retroactively invalidate every generalisation claim built on them, including MoorAI's own.
The split algorithm is published and needs no stored seed, so any vendor can produce their own from their own corpus: bucket by stratum, sort by id inside the bucket, assign alternately — even index to tune, odd to test — then never open the test half while tuning. No random number generator, so no seed to record, lose or quietly change.
MoorAI scores 100% recall (61/61) on heldout-v2-tune — the half its detectors were developed against. That figure measures memorisation, not generalisation, and it is in the repository precisely so the gap between a tune-half score and a locked-half score is visible rather than flattering.
An adapter is a single ES module. Only scanText is required. Optional scanAction, scanSession and scanEvents unlock the action, sequence and event-graph samples — and you are not penalised for omitting them: samples whose harness you do not implement land in a not-applicable bucket and are excluded from every rate, so a text-only product does not appear to have missed samples it was never shown.
# clone, then run the naive reference adapter to see the output shape git clone https://github.com/gitayg/agentic-security-benchmark cd agentic-security-benchmark npm run score -- --adapter keyword # then score your own node scorers/run.mjs --adapter ./acme-adapter.mjs --corpus all --json > results/acme.json
A finding with no action is treated as notify, which is a detection and never a prevention. Prevention is derived from the enforcement action your own policy would take — justify, block or kill — never from the fact that something fired. You cannot earn a prevented outcome by returning more findings. If your policy would only log, your prevention rate correctly reads low; that is the number, not a penalty.
Missing id, or an action outside the allowed list, fails the run with the adapter name, the method and the offending index. Nothing is coerced or silently dropped, because a number produced from a swallowed error is a number nobody can reproduce.
Corpora only get better if people who are not the maintainer add to them. There are structured issue forms for proposing an attack sample, proposing a benign sample, reporting a mislabelled sample, and submitting results for another product — see CONTRIBUTING.md. Additions to a held-out split are not accepted from outside, because that would compromise the split.
The results directory holds one file per product and is built to hold more than one. Every run currently in it was executed by the benchmark maintainer, who is also the author of MoorAI — the MoorAI result says so in its own conflictOfInterest field. These are not independent tests. No third-party lab has reproduced anything here.
Two of them exist to make the third readable: a null adapter that detects nothing, so the floor of every rate is visible, and a naive keyword adapter, so a score can be compared against something rather than against nothing.
Every scored run, with its per-vector breakdown, is on the leaderboard. A vendor who wants a row opens a pull request against the benchmark repository.
The repository keeps this list in its own README. It is reproduced here rather than summarised away.
Not the benchmark, not its corpora, not its harness, not any result published in it. The Guidelines are cited as published criteria this benchmark grades against. That is self-assessment, not accreditation. AMTSO is a trademark of the Anti-Malware Testing Standards Organization.
No sample is a real incident and no deployment telemetry feeds them. What a result describes is behaviour against attacks somebody knew how to write.
The first result was produced by the author of both the product and the benchmark. It is not an independent test and is not presented as one.
This harness is deterministic and model-free by design, so model-refusal and model-recognition are reported as measured-as-absent zeros. Every recall figure in the results directory is a raw detection rate, not a marginal contribution — including MoorAI's — and must not be compared against a figure that has had a baseline subtracted.
It keys off expectThreat, which holds one product's threat ids, so it is reported as n/a for adapters that do not share that id space. A vendor-neutral right-reason label is an open problem; proposals are welcome.
Nine samples in benign-web-content-tune are deliberately labelled as attacks: prompt-injection tutorial pages, on the reasoning that an injection example on a tutorial page is a live payload — the agent fetching it cannot tell teaching material from an attack, because the text is identical either way. The file's own header comment is stale on this point, and that is written down rather than quietly fixed.
For the one adapter that implements the action surface, vector 4 is measured by driving the product's real enforcement hook — but at its built-in default, with no enrolled tenant policy. An enforcing policy takes the same hook from 18 of 57 actions stopped to 43 of 57. Prevention is understated by about that much, and the parameter is documented rather than assumed.
The locked test halves are withheld on purpose. See section 04.
Sample counts on this page are read from the benchmark's own scored output, pinned to commit 4bc3735e of gitayg/agentic-security-benchmark; they move as corpora grow. The benchmark is Apache-2.0. The corpora and harness originated in the MoorAI agent, which is AGPL-3.0 and is a separate project — nothing in the benchmark imposes AGPL obligations on a product under test. AMTSO has not reviewed, certified or endorsed this benchmark, MoorAI, or any result published here; AMTSO is a trademark of the Anti-Malware Testing Standards Organization. OWASP® is a trademark of the OWASP Foundation; ATT&CK® and ATLAS™ are trademarks of The MITRE Corporation. The attack-family taxonomy the family-keyed corpora reference is HackAgent.