hackwither_

evidential ceiling / benchmarks / strongreject · n = 313 · §5.2, Fig. 2

StrongREJECT: what a clean run can certify

StrongREJECT (Souly et al. 2024) has 313 prompts. In the paper’s audit its size is the only input that is specific to it, so what it can certify follows from n alone. Each row is the smallest harm rate pu at which a clean run, taken as independent trials at r = 0.5, reaches that claim.

What it can certify

claima clean run reaches it for p_u ≥
Harm rate bounded above at 95% confidenceany p_u · bound 0.95% (1 in 105)
Belief shifts 2× toward acceptable risk τ = 0.54.41 × 10⁻³
False certification held below 5%9.53 × 10⁻³
Belief shifts 10× τ = 0.11.46 × 10⁻²
Belief shifts 100× τ = 0.012.88 × 10⁻²

a benchmark below the ceiling is not a bad benchmark; it is being asked a question its size cannot answer

At the paper’s worked example

Benchmark: StrongREJECT, n = 313 · pu = 10⁻³ · r = 0.5 · ps = 5 × 10⁻⁴ · τ = 0.1 · independent trials

clean-sheet evidence
0.226 bits
likelihood ratio Λ0
0.8551.17× shift in odds
reaches τ = 0.1 for pu ≥
1.46 × 10⁻²pmin at n = 313
crossing rate p×
2.21 × 10⁻³above it, a clean sheet outweighs one harm
95% upper bound, clean run
0.95%1 in 105, Clopper–Pearson

A clean run of StrongREJECT’s 313 prompts, taken as independent trials, gives 0.226 bits of evidence that the harm rate is 5 × 10⁻⁴ rather than 10⁻³. At τ = 0.1 (a 10× shift), a clean run this size reaches the threshold only for harm rates of 1.46 × 10⁻² or more.

2×10×100×1,000×τ = 0.1

0.226 bits toward acceptable riskTicks mark the shift in odds; each bit doubles them.

Required trials against harm rate, to scaleAt p_u = 10⁻³, reaching τ = 0.1 takes 4,602 clean trials; a clean run of 313 reaches it only for p_u of 1.46 × 10⁻² or more. The crossing rate p× is 2.21 × 10⁻³.10⁻⁴10⁻³10⁻²10⁻¹110¹10²10³10⁴10⁵10⁶harm rate p_u (log) →trials (log)n = 313p_minp×p_utrials needed, τ = 0.1
Fig. 1 To scale for the parameters above. Trials a clean run needs to reach τ = 0.1, against harm rate (r = 0.5). Shaded: reachable within this run’s n. Hatched: below pmin = 1.46 × 10⁻², where no clean run that size reaches τ. Pink: your pu, with the gap to the trials it needs. Open circle: the crossing rate p×.

pu = 10⁻³ is the rate the paper’s AdvBench example uses (§4.1). It is not an estimate for StrongREJECT; try your own

Cite

Kaur, B. (2026). What AI Red-Team Evaluations Can and Cannot Prove. arXiv:2607.21735, §5.2, Fig. 2. This page: hackwither.co.in/evidential-ceiling/b/strongreject/, Evidential Ceiling Calculator v1.0.0.

Suite ids are frozen once shipped, so this is a permanent link you can cite.