hackwither_

evidential ceiling / benchmarks / advbench · n = 520 · §5.1, Fig. 2

AdvBench: what a clean run can certify

AdvBench (Zou et al. 2023) has 520 prompts. In the paper’s audit its size is the only input that is specific to it, so what it can certify follows from n alone. Each row is the smallest harm rate pu at which a clean run, taken as independent trials at r = 0.5, reaches that claim.

open in the calculator·the paper·all eight suites

What it can certify

claima clean run reaches it for p_u ≥
Harm rate bounded above at 95% confidenceany p_u · bound 0.57% (1 in 174)
Belief shifts 2× toward acceptable risk τ = 0.52.66 × 10⁻³
False certification held below 5%5.74 × 10⁻³
Belief shifts 10× τ = 0.18.8 × 10⁻³
Belief shifts 100× τ = 0.011.75 × 10⁻²

a benchmark below the ceiling is not a bad benchmark; it is being asked a question its size cannot answer

At the paper’s worked example

Benchmark: AdvBench, n = 520 · pu = 10⁻³ · r = 0.5 · ps = 5 × 10⁻⁴ · τ = 0.1 · independent trials

clean-sheet evidence
0.375 bits
likelihood ratio Λ0
0.7711.3× shift in odds
reaches τ = 0.1 for pu ≥
8.8 × 10⁻³pmin at n = 520
crossing rate p×
1.33 × 10⁻³above it, a clean sheet outweighs one harm
95% upper bound, clean run
0.57%1 in 174, Clopper–Pearson

A clean run of AdvBench’s 520 prompts, taken as independent trials, gives 0.375 bits of evidence that the harm rate is 5 × 10⁻⁴ rather than 10⁻³. At τ = 0.1 (a 10× shift), a clean run this size reaches the threshold only for harm rates of 8.8 × 10⁻³ or more.

2×10×100×1,000×τ = 0.1

0.375 bits toward acceptable riskTicks mark the shift in odds; each bit doubles them.

Required trials against harm rate, to scaleAt p_u = 10⁻³, reaching τ = 0.1 takes 4,602 clean trials; a clean run of 520 reaches it only for p_u of 8.8 × 10⁻³ or more. The crossing rate p× is 1.33 × 10⁻³.10⁻⁵10⁻⁴10⁻³10⁻²10⁻¹10¹10²10³10⁴10⁵10⁶harm rate p_u (log) →trials (log)n = 520p_minp×p_utrials needed, τ = 0.1
Fig. 1 To scale for the parameters above. Trials a clean run needs to reach τ = 0.1, against harm rate (r = 0.5). Shaded: reachable within this run’s n. Hatched: below pmin = 8.8 × 10⁻³, where no clean run that size reaches τ. Pink: your pu, with the gap to the trials it needs. Open circle: the crossing rate p×.

pu = 10⁻³ is the rate the paper’s AdvBench example uses (§4.1). It is not an estimate for AdvBench; try your own

Cite

Kaur, B. (2026). What AI Red-Team Evaluations Can and Cannot Prove. arXiv:2607.21735, §5.1, Fig. 2. This page: hackwither.co.in/evidential-ceiling/b/advbench/, Evidential Ceiling Calculator v1.0.0.

Suite ids are frozen once shipped, so this is a permanent link you can cite.