evidential ceiling / benchmarks · 8 suites · paper §5
Audited benchmarks: what each can certify
The eight evaluation suites audited in the paper. The only suite-specific input is size, so these are properties of a design at an assumed operating point, not a ranking of benchmark quality. Each column is the smallest harm rate at which a clean run reaches the claim, at r = 0.5.
The suites
| suite | n | 2× | FC < 5% | 10× | 100× |
|---|---|---|---|---|---|
| AdvBench | 520 | 2.66 × 10⁻³ | 5.74 × 10⁻³ | 8.8 × 10⁻³ | 1.75 × 10⁻² |
| HarmBench | 400 | 3.46 × 10⁻³ | 7.46 × 10⁻³ | 1.14 × 10⁻² | 2.26 × 10⁻² |
| StrongREJECT | 313 | 4.41 × 10⁻³ | 9.53 × 10⁻³ | 1.46 × 10⁻² | 2.88 × 10⁻² |
| WildGuard | 1,725 | 8.03 × 10⁻⁴ | 1.74 × 10⁻³ | 2.66 × 10⁻³ | 5.32 × 10⁻³ |
| SORRY-Bench | 440 | 3.14 × 10⁻³ | 6.79 × 10⁻³ | 1.04 × 10⁻² | 2.06 × 10⁻² |
| XSTest | 250 | 5.52 × 10⁻³ | 1.19 × 10⁻² | 1.82 × 10⁻² | 3.58 × 10⁻² |
| SafetyBench | 2,100 | 6.6 × 10⁻⁴ | 1.43 × 10⁻³ | 2.19 × 10⁻³ | 4.37 × 10⁻³ |
| HEx-PHI | 330 | 4.19 × 10⁻³ | 9.04 × 10⁻³ | 1.38 × 10⁻² | 2.73 × 10⁻² |
2×, 10×, 100×: belief shifts toward acceptable risk by that factor (τ = 0.5, 0.1, 0.01). FC < 5%: false certification held below five percent. Sources: paper §5, Fig. 2.