SafetyBench: what a clean run can certify

claim	a clean run reaches it for p_u ≥
Harm rate bounded above at 95% confidence	any p_u · bound 0.14% (1 in 701)
Belief shifts 2× toward acceptable risk τ = 0.5	6.6 × 10⁻⁴
False certification held below 5%	1.43 × 10⁻³
Belief shifts 10× τ = 0.1	2.19 × 10⁻³
Belief shifts 100× τ = 0.01	4.37 × 10⁻³

SafetyBench is a multiple-choice knowledge probe; n = 2,100 is its Chinese subset, 300 questions in each of seven categories (§5).

Benchmark: SafetyBench, n = 2,100 · p_u = 10⁻³ · r = 0.5 · p_s = 5 × 10⁻⁴ · τ = 0.1 · independent trials

clean-sheet evidence

1.516 bitsclosed form 1.515

likelihood ratio Λ₀

0.352.86× shift in odds

reaches τ = 0.1 for p_u ≥

2.19 × 10⁻³p_min at n = 2,100

crossing rate p_×

3.3 × 10⁻⁴above it, a clean sheet outweighs one harm

95% upper bound, clean run

0.14%1 in 701, Clopper–Pearson

A clean run of SafetyBench’s 2,100 prompts, taken as independent trials, gives 1.516 bits of evidence that the harm rate is 5 × 10⁻⁴ rather than 10⁻³. At τ = 0.1 (a 10× shift), a clean run this size reaches the threshold only for harm rates of 2.19 × 10⁻³ or more.