04 — testing & measurement

Adversarial mutation

Mechanically generating fresh variants of known attacks along defined transformation axes — synonym and voice swaps, encodings, homoglyphs, letter-spacing, embedding the directive in a code comment or JSON blob, persona recombinations — to test generalisation rather than memory. Validity is the hard part: a transformation that garbles the payload into nonsense is a broken sample, not a missed detection, so every mutation has to be invertible and round-tripped back to the original before it is allowed to be scored.

See also overfitting · h4rm3l · AutoDAN

Related terms

Overfitting Testing & measurement
Fitting the examples instead of the phenomenon. In detection work it usually looks like a rule matched to the exact phrasings in the corpus — a pattern…
h4rm3l Attack families
A family built around composable string transformations: a small language of primitives — character substitution, insertion, role framing, encoding — that…
AutoDAN Attack families
The automated successor to DAN: instead of a human writing the persona prompt, a search process — typically a genetic algorithm or gradient-guided…

Testing & measurement

The words that decide whether a published detection number is evidence or decoration. Most vendor numbers in this space are quoted without any of them; see how we test for how these are applied in practice.

Adversarial mutation is term 14 of 15 in this part of the glossary.

Held-out set Tune/test split Overfitting Baseline validation False positive False negative Precision Recall Hard negative Detection vs prevention Model refusal Marginal value Red teamingAdversarial mutationVariance

This page is one entry from the agentic AI security glossary, which defines 60 terms in the same style. Where a term belongs to a published taxonomy or standard — the HackAgent attack families, the OWASP lists, AMTSO's guidelines, MITRE ATLAS, ISO/IEC 42001, the NIST AI RMF, the EU AI Act — the authoritative wording is the source document's, not ours, and specifics should be verified there. "Lethal trifecta" is Simon Willison's term. OWASP® is a trademark of the OWASP Foundation; ATT&CK® and ATLAS™ are trademarks of The MITRE Corporation. Naming a framework here is description, not a claim of certification or endorsement.