04 — testing & measurement

Red teaming

Adversarial testing that tries to make a system fail, rather than confirming it works. For AI systems it spans manual probing, curated attack corpora, and automated attack generation. Its weakness is repeatability: a human red-teamer's findings are hard to re-run after a change, which is why serious programmes convert findings into a scored, deterministic corpus that regressions can be measured against.

Related terms

Adversarial mutation Testing & measurement
Mechanically generating fresh variants of known attacks along defined transformation axes — synonym and voice swaps, encodings, homoglyphs…
MITRE ATLAS Frameworks & standards
ATT&CK for machine learning: a knowledge base of adversary tactics and techniques against AI systems, grounded in real incidents and published research…

Testing & measurement

The words that decide whether a published detection number is evidence or decoration. Most vendor numbers in this space are quoted without any of them; see how we test for how these are applied in practice.

Red teaming is term 13 of 15 in this part of the glossary.

Held-out set Tune/test split Overfitting Baseline validation False positive False negative Precision Recall Hard negative Detection vs prevention Model refusal Marginal valueRed teamingAdversarial mutation Variance

This page is one entry from the agentic AI security glossary, which defines 60 terms in the same style. Where a term belongs to a published taxonomy or standard — the HackAgent attack families, the OWASP lists, AMTSO's guidelines, MITRE ATLAS, ISO/IEC 42001, the NIST AI RMF, the EU AI Act — the authoritative wording is the source document's, not ours, and specifics should be verified there. "Lethal trifecta" is Simon Willison's term. OWASP® is a trademark of the OWASP Foundation; ATT&CK® and ATLAS™ are trademarks of The MITRE Corporation. Naming a framework here is description, not a claim of certification or endorsement.