hackwither_

evidential ceiling / benchmarks / sorry-bench · n = 440 · Fig. 2

SORRY-Bench: what a clean run can certify

SORRY-Bench (Xie et al. 2024) has 440 prompts. In the paper’s audit its size is the only input that is specific to it, so what it can certify follows from n alone. Each row is the smallest harm rate pu at which a clean run, taken as independent trials at r = 0.5, reaches that claim.

open in the calculator·the paper·all eight suites

What it can certify

claima clean run reaches it for p_u ≥
Harm rate bounded above at 95% confidenceany p_u · bound 0.68% (1 in 147)
Belief shifts 2× toward acceptable risk τ = 0.53.14 × 10⁻³
False certification held below 5%6.79 × 10⁻³
Belief shifts 10× τ = 0.11.04 × 10⁻²
Belief shifts 100× τ = 0.012.06 × 10⁻²

a benchmark below the ceiling is not a bad benchmark; it is being asked a question its size cannot answer

At the paper’s worked example

Benchmark: SORRY-Bench, n = 440 · pu = 10⁻³ · r = 0.5 · ps = 5 × 10⁻⁴ · τ = 0.1 · independent trials

clean-sheet evidence
0.318 bitsclosed form 0.317
likelihood ratio Λ0
0.8021.25× shift in odds
reaches τ = 0.1 for pu ≥
1.04 × 10⁻²pmin at n = 440
crossing rate p×
1.58 × 10⁻³above it, a clean sheet outweighs one harm
95% upper bound, clean run
0.68%1 in 147, Clopper–Pearson

A clean run of SORRY-Bench’s 440 prompts, taken as independent trials, gives 0.318 bits of evidence that the harm rate is 5 × 10⁻⁴ rather than 10⁻³. At τ = 0.1 (a 10× shift), a clean run this size reaches the threshold only for harm rates of 1.04 × 10⁻² or more.

2×10×100×1,000×τ = 0.1

0.318 bits toward acceptable riskTicks mark the shift in odds; each bit doubles them.

Required trials against harm rate, to scaleAt p_u = 10⁻³, reaching τ = 0.1 takes 4,602 clean trials; a clean run of 440 reaches it only for p_u of 1.04 × 10⁻² or more. The crossing rate p× is 1.58 × 10⁻³.10⁻⁴10⁻³10⁻²10⁻¹110¹10²10³10⁴10⁵10⁶harm rate p_u (log) →trials (log)n = 440p_minp×p_utrials needed, τ = 0.1
Fig. 1 To scale for the parameters above. Trials a clean run needs to reach τ = 0.1, against harm rate (r = 0.5). Shaded: reachable within this run’s n. Hatched: below pmin = 1.04 × 10⁻², where no clean run that size reaches τ. Pink: your pu, with the gap to the trials it needs. Open circle: the crossing rate p×.

pu = 10⁻³ is the rate the paper’s AdvBench example uses (§4.1). It is not an estimate for SORRY-Bench; try your own

Cite

Kaur, B. (2026). What AI Red-Team Evaluations Can and Cannot Prove. arXiv:2607.21735, Fig. 2. This page: hackwither.co.in/evidential-ceiling/b/sorry-bench/, Evidential Ceiling Calculator v1.0.0.

Suite ids are frozen once shipped, so this is a permanent link you can cite.