Every detection number on this site comes from a harness in the open-source agent repository. This page names the taxonomy those corpora are keyed to, the grading standard we grade ourselves against, the one model we run, each harness and what it actually measures — and then the limits, because a methodology page that only lists strengths is marketing with footnotes.
The short version of why this page exists: a coverage percentage without a corpus, a false-positive rate, a held-out set and a refusal baseline is a number about nothing. We would rather publish a smaller figure that survives inspection than a larger one that does not. Terms used here are defined in the glossary.
Three external things, and only three: a taxonomy, a grading standard, and a local model. Everything else in the measurement stack is code in the repository, described in section 03.
The published attack-family taxonomy our red-team corpora are keyed to — ten families: CipherChat, FlipAttack, h4rm3l, DAN, AutoDAN, BoN, AdvPrefix, PAP, PAIR and TAP. Every attack sample we score carries the family it belongs to, so coverage can be reported per family rather than as one averaged figure that hides which half of the taxonomy is uncovered. We use the taxonomy — the family definitions and their published techniques — as the labelling scheme for corpora we generate ourselves. Source: AISecurityLab/hackagent ↗. Each family is defined in the glossary.
The Anti-Malware Testing Standards Organization's Guidelines for Testing of Agentic Security Products v1.0 ↗, published 2 September 2026 — the first testing standard written for this product category. We use it as the grading criteria: its six test-case classification dimensions (attack vector, target of protection, environment type, harm type, severity with disclosed criteria, required capability), its rule that model refusal must not be counted as product detection or prevention, and its request for a distribution of outcomes rather than a single pass or fail. AMTSO has not reviewed, certified or endorsed MoorAI — we read their document and graded ourselves against it.
One model, run locally through Ollama: llama3:latest — 8.0B parameters, Q4_0 quantization, digest 365c0bd3c000. It plays two roles: the target whose refusal baseline we measure per attack family, and the judge in the optional semantic escalation path. It is explicitly a local proxy, not a frontier model, and this matters for reading our results — see limits. Running it locally is deliberate: the corpora never leave the machine, and the measurement costs nothing per run, which is what makes repeated-run variance affordable at all.
All authored in-repo and versioned alongside the code, so any number here can be re-run: a family-keyed HackAgent set, a first held-out set (now burned by tuning and retired as a generalisation measure — see discipline), a mutation-generated set split into tune and locked halves, a 610-prompt benign corpus with 269 adversarially-shaped hard negatives, and four dedicated per-vector corpora for the tool/MCP supply chain (72 attacks, 25 benign), memory / cross-agent propagation (42 attacks, 25 benign), outbound action (57 attacks, 24 benign) and indirect content (45 attacks, 17 benign).
Listing tools you did not run is the whole point of a methodology page, so here it is explicitly rather than by omission.
We have not used garak, PyRIT, promptfoo, Burp Suite, Metasploit, or any other third-party penetration-testing or LLM red-teaming tool to produce any number published on this site. Not one figure here came from a scan we ran with somebody else's scanner. Where a well-known tool name would make a claim sound more credible, its absence from our stack is the honest fact, and we would rather say so than let a logo imply a test that never happened.
Nor have we been tested by an independent lab. Every measurement below is self-run, on our own corpora, on our own hardware. That is a real limitation, not a formality — see limits — and it is why the harnesses are in a public AGPL-3.0 repository rather than described in a PDF: the falsifiable version of "trust us" is "re-run it".
Eleven measurement scripts in the agent repository. Each one is described by what it measures, because a harness that cannot state what would falsify it is not a measurement. All of them import the shipped detection engine rather than re-implementing it, and none of them mutates a detector or a corpus.
| Harness | What it measures |
|---|---|
| Red-team coverage evalscripts/redteam-eval.mjs | Detection coverage across the ten HackAgent families, deterministically and LLM-free: every sample carries a ground-truth shouldDetect label, so coverage, precision and recall are computed by comparing the engine's verdict to that label — no model, no judge, reproducible on any machine. An optional flag routes samples through the on-device model layer instead; the default run never imports it, so the headline number is never model-assisted. |
| Locked tune/test splitterscripts/split-heldout-v2.mjs | Splits a corpus once into a tune half and a locked test half, stratified by family and transformation axis so both halves carry the same mix — an unstratified split could hand one half every homoglyph sample and make the result meaningless. Within each bucket samples are sorted by id and assigned alternately, so the split is reproducible from the corpus alone with no stored seed. Tuning work may see only the tune half. |
| Adversarial mutation generatorscripts/mutate-attacks.mjs | Generates a fresh attack set by transforming each family's published technique along explicit axes — synonym and voice swaps, directive-in-code-comment / JSON / blockquote indirection, homoglyphs, letter-spacing, Caesar and ROT13, persona and persuasion recombinations. It deliberately does not read the detectors: steering mutations toward what the engine already catches would burn the set. Every transform except a bounded single-character typo axis is information-preserving and reversible, so the malicious core is recoverable and the sample is provably still an attack. |
| Closed-loop adversaryscripts/closed-loop-mutate.mjs | Models an adversary who learns: round 0 is the plaintext cores, round 1 every single-step transformation, and round N+1 every one-step extension of only the chains that evaded in round N. A one-shot generator understates risk because it never composes two evasions; this spends its budget only on productive directions and reports which transformation chains survive. Validity is mechanical, not asserted — every candidate is rendered backwards and must reproduce the malicious core character for character, or it is dropped before it is ever scored. |
| Benign false-positive scorerscripts/score-benign-v2.mjs | The other half of the question recall never answers: across 610 realistic benign developer and agent prompts, 269 of them adversarially shaped hard negatives, how often does the engine fire when it should stay silent? Reports the false-positive rate per bucket and per hard-negative twin family, and for every false positive the detector and threat ids that fired. It reuses the same reducer as the recall benchmarks, so the two numbers are apples-to-apples. |
| Model-refusal baselinescripts/measure-refusal-baseline.mjs | Per attack family, how often the model refuses on its own with no product involved — run N times because a single run is a point estimate. This is what converts a raw coverage figure into marginal value: recall added where the model already refuses is worth close to nothing. Its most useful output has been negative — it showed that an entire detection wave we had shipped bought almost no marginal protection, because the model refused those families anyway. |
| Variance harnessscripts/measure-variance.mjs | Repeated runs of the same scored set, reporting the distribution and per-sample stability rather than one number. It exists because we watched model-assisted figures move between runs — a judge recovering eight attacks once and five the next time, a benign false-positive count drifting by one. AMTSO asks for the distribution of outcomes rather than hiding variability behind a single pass or fail; this produces it. Measurement only: it mutates no detector, no corpus and no other script. |
| Action-layer blocking validatorscripts/moorai-validate-blocking.mjs | The one that tests prevention rather than detection. It assumes the agent has already lost — hijacked, jailbroken, whatever — and asks whether the malicious tool call it then attempts is denied. For each action it feeds a real PreToolUse payload to the real hook as a subprocess and records the actual decision from its stdout; it does not re-implement the hook's logic, which is the entire point. This is the harness behind any claim that uses the word "blocked". |
| Per-vector scorersscripts/score-vectors.mjs | Scores the two AMTSO attack vectors that ship detectors but had never been measured: tool / skill / extension / MCP supply chain and memory, context and cross-agent propagation. They need their own scorer because neither fits "one sample = one string": supply-chain samples are tool metadata and config files only reachable at the tool, file and index stages, and cross-agent samples are sequences — written in one session, consumed in a later one — or event-graph shapes with no text at all. Optimistic and strict readings are reported separately and are not interchangeable. |
| Cross-agent end-to-end reachabilityscripts/score-vector5-production.mjs | Whether a real deployment can actually see the cross-agent detections, as opposed to whether the library function returns a finding. It refuses to call the library at all: it seeds the on-device event window and then drives the two paths a deployment really has — the real PreToolUse hook, and the real agentwatch emitter — counting only what arrives at a stand-in management server. It was written because the library scored full marks on a capability nothing on the enforcement path was calling. |
| AMTSO dimension-coverage reporterscripts/amtso-coverage.mjs | Classifies every corpus sample along AMTSO's six dimensions and reports the distribution, flagging zero-coverage cells by name rather than reporting an average. Labels live in a sidecar; the corpora are never modified. AMTSO's own values are used verbatim and any local extension is prefixed and reported separately so it can never be mistaken for AMTSO coverage. Its job is to find the holes in our testing, and it does — see limits. |
Anyone can write a scoring script. The five rules below are what make its output worth reading, and each of them has cost us a number we would have preferred to publish.
A corpus is split once, stratified, and the test half is off-limits to detector work — not "we tried not to look at it", but a separate file that the tuning wave does not open. We enforce this because we broke it once: an earlier held-out set was tuned against and thereby destroyed as a generalisation measure, and it has been retired rather than quietly re-used. The gap between the tune half's score and the locked half's score is published as the overfit margin, because that gap is the finding.
Blocking something that was never going to succeed is not protection. Two baselines run underneath the coverage numbers: the refusal baseline, which removes credit for anything the model would have refused anyway, and the enforcement baseline, which confirms an action actually executes when the product is not in the path. AMTSO states the refusal principle; the practical consequence is a priority rule we now follow — measure the refusal baseline for a family first, and build only where the model complies.
Every recall figure has a precision figure measured on the same engine against the 610-prompt benign corpus, and the two are never combined into a single flattering score. The corpus is deliberately hostile to us: 269 of its 610 samples are hard negatives — legitimate security questions, benign role-play, urgent-sounding ordinary requests, exfiltration-shaped but innocent prompts, and 101 deliberately obfuscated benign samples (base64 blobs, minified JavaScript, non-English prose, leetspeak identifiers, ASCII tables) added because the previous corpus contained exactly one — grouped into twin families so we can see which kind of benign prompt we misfire on, not just how often.
A test that has never been observed failing is not evidence; it is a green light with no wiring behind it. So the production path is deliberately broken, the test is confirmed to go red, the break is reverted, and what was broken is recorded. This is the rule that catches the embarrassing class of bug — the harness that scores a capability nothing in the shipping product calls, which is exactly what the cross-agent end-to-end reachability harness was written to expose after a library-level score of full marks turned out to be unreachable in a real deployment.
Anything involving a model is a sample from a distribution. A single passing run is a point estimate presented as a fact, and we have watched those estimates move — which is why model-assisted figures are run N times and published as a spread, and why the deterministic, LLM-free number is always the headline while the model-assisted one is reported separately and never merged into it.
Stated plainly, because a limit you have to discover for yourself is a limit that was hidden. These are the four that would most change how a result should be read.
Prevention — the action-layer blocking numbers — is measured against a single agent host, because that is where a PreToolUse hook exists to enforce at. Other hosts are covered at the detection layer, and a detection claim is not an enforcement claim. Any statement that a malicious tool call was blocked should be read as scoped to the host it was measured on.
The refusal baseline and the semantic judge both run on llama3:latest (8.0B, Q4_0). A frontier model refuses more, and decodes obfuscated payloads far better. Inspecting our own runs showed the 8B model was frequently not complying with obfuscated attacks so much as failing to decode them — it read base64 and hallucinated the plaintext, misread leetspeak as a cipher, botched reversals. A frontier model would decode those correctly and might then refuse, which means a meaningful share of the marginal value we measure for the obfuscation families is soft, and we say so rather than banking it.
Labelling every classified attack sample against the AMTSO taxonomy gives 624 samples, of which 296 (47.4%) are vector 1, direct input — 130 are vector 2 (indirect content), 78 vector 3 (tool / MCP supply chain), 62 vector 4 (outbound action), 42 vector 5 (memory / cross-agent propagation), and 16 are static code artifacts carrying no vector. All four of the non-direct vectors now have dedicated, sub-technique-enumerated corpora and their own scorers, so no vector is starved; vector 1 is still the largest single share by a wide margin, because it remains the easiest to author. The reporter prints the distribution every run.
The sharper limit is inside vector 2. The dedicated set scores 73.3% recall at 78.6% precision, but only 11 of the 33 catches fire for the right reason — the rest are caught incidentally by a detector aimed at something else, and an incidental catch does not survive the attacker changing one thing. data-poisoning-fact, where poisoned content changes the agent's answer while issuing no directive at all, sits at 25%. Underneath that is a structural problem rather than a rule gap: inbound WebFetch content cannot be scanned from a PreToolUse hook, because the page does not exist yet when the hook fires — and that is the delivery path for indirect prompt injection, the defining agentic attack.
Every figure here is self-run, on self-authored corpora, on our own hardware. No third-party lab has verified, reproduced or audited any of it. The mitigation we can offer is not a certificate but a repository: the agent, the harnesses and the corpora are open source under AGPL-3.0, so the falsifiable version of this page is somebody re-running it and disagreeing.
We read AMTSO's published Guidelines for Testing of Agentic Security Products v1.0 and graded our own testing against its criteria. That is self-assessment, not accreditation. AMTSO has not reviewed our methodology, has not tested our product, has not certified anything, and has no relationship with us. Nothing on this page is an AMTSO result, an AMTSO score, or an AMTSO endorsement, and any reading of it as one is wrong. The same applies to OWASP, MITRE and every other body named here.
Harness paths refer to the open-source MoorAI agent repository (AGPL-3.0). Corpus sizes and the AMTSO dimension distribution above were produced by running the in-repo reporter; they move as corpora grow, so treat them as of the current release rather than as fixed constants. AMTSO has not reviewed, certified or endorsed MoorAI, and the Guidelines are cited as published criteria we graded ourselves against. OWASP® is a trademark of the OWASP Foundation; ATT&CK® and ATLAS™ are trademarks of The MITRE Corporation; AMTSO is a trademark of the Anti-Malware Testing Standards Organization. Llama 3 is used under the Meta Llama 3 Community License. No third-party penetration-testing or LLM red-teaming tool was used to produce any figure referenced on this page.