i red-team AI systems for a living, and last week i told NIST to be more realistic about what a red-team pass actually proves.
NIST put out a draft framework called TEVV-Athlon (AI 200-2). it's a recipe for building your own AI evaluation in four stages instead of one fixed test, and red teaming is one of the ways it collects evidence. it was open for public comment until Oct 6, so i wrote in.
run 520 red-team prompts against a model that fails once in every 1,000 interactions, and there's a 59% chance you see zero failures. zero. a clean run of 520 prompts can only tell you the failure rate is below roughly 1 in 175. that's the ceiling, and no amount of writing it up neatly moves it.
now look at where that lands. the draft tells evaluators to summarise violations with descriptive statistics. so a result of "0 violations" gets passed up the chain, and a decision-maker reads it as "safe". but 0 out of 520 doesn't mean safe. it means the test was too small to see the failure it was looking for.
my ask was simple: every red-team result should ship with its evidential ceiling. i asked NIST to require a short "statement of evidential strength" on every result that counts failures: the number of trials, the number of failures, the upper bound those numbers actually support, and whether the test was even big enough for the risk you care about. it turns "0 violations" from a verdict back into what it is, a measurement with a limit.
none of this is new maths. it's the same binomial arithmetic drug trials have used for decades. we just haven't been writing it down for AI.
i put the full comment, the numbers, and a calculator online if you want to poke at it:
- the comment: what i sent NIST
- the calculator: try 520 prompts, 0 failures
- the paper it's built on: what AI red-team evaluations can and cannot prove
as always, happy hacking!!
–HackWitHer