Overfitting
Fitting the examples instead of the phenomenon. In detection work it usually looks like a rule matched to the exact phrasings in the corpus — a pattern that catches "override your safety rules" but not "override your safety guidelines" — producing an excellent in-sample score and a much worse one on anything new. It is not a hypothetical: a corpus scoring near-perfect in-sample can drop to roughly a third on fresh mutations of the same attacks, and that drop is the real result.
See also held-out set · adversarial mutation · leetspeak
Related terms
-
Held-out set
Testing & measurement
Test samples deliberately kept away from whoever is building the detection, so the score answers "does this generalise?" rather than "does this…
-
Adversarial mutation
Testing & measurement
Mechanically generating fresh variants of known attacks along defined transformation axes — synonym and voice swaps, encodings, homoglyphs…
-
Leetspeak
Attack concepts
Substituting digits and symbols for visually similar letters — 3 for e, 1 or ! for i/l, 4 or @ for a, 0 for o — as in h4rm3l. Models read it fluently;
Testing & measurement
The words that decide whether a published detection number is evidence or decoration. Most vendor numbers in this space are quoted without any of them; see how we test for how these are applied in practice.