hackwither_

evidential ceiling / benchmarks / hex-phi · n = 330 · Fig. 2

HEx-PHI: what a clean run can certify

HEx-PHI (Qi et al. 2023) has 330 prompts. In the paper’s audit its size is the only input that is specific to it, so what it can certify follows from n alone. Each row is the smallest harm rate pu at which a clean run, taken as independent trials at r = 0.5, reaches that claim.

open in the calculator·the paper·all eight suites

What it can certify

claima clean run reaches it for p_u ≥
Harm rate bounded above at 95% confidenceany p_u · bound 0.9% (1 in 111)
Belief shifts 2× toward acceptable risk τ = 0.54.19 × 10⁻³
False certification held below 5%9.04 × 10⁻³
Belief shifts 10× τ = 0.11.38 × 10⁻²
Belief shifts 100× τ = 0.012.73 × 10⁻²

a benchmark below the ceiling is not a bad benchmark; it is being asked a question its size cannot answer

At the paper’s worked example

Benchmark: HEx-PHI, n = 330 · pu = 10⁻³ · r = 0.5 · ps = 5 × 10⁻⁴ · τ = 0.1 · independent trials

clean-sheet evidence
0.238 bits
likelihood ratio Λ0
0.8481.18× shift in odds
reaches τ = 0.1 for pu ≥
1.38 × 10⁻²pmin at n = 330
crossing rate p×
2.1 × 10⁻³above it, a clean sheet outweighs one harm
95% upper bound, clean run
0.9%1 in 111, Clopper–Pearson

A clean run of HEx-PHI’s 330 prompts, taken as independent trials, gives 0.238 bits of evidence that the harm rate is 5 × 10⁻⁴ rather than 10⁻³. At τ = 0.1 (a 10× shift), a clean run this size reaches the threshold only for harm rates of 1.38 × 10⁻² or more.

2×10×100×1,000×τ = 0.1

0.238 bits toward acceptable riskTicks mark the shift in odds; each bit doubles them.

Required trials against harm rate, to scaleAt p_u = 10⁻³, reaching τ = 0.1 takes 4,602 clean trials; a clean run of 330 reaches it only for p_u of 1.38 × 10⁻² or more. The crossing rate p× is 2.1 × 10⁻³.10⁻⁴10⁻³10⁻²10⁻¹110¹10²10³10⁴10⁵10⁶harm rate p_u (log) →trials (log)n = 330p_minp×p_utrials needed, τ = 0.1
Fig. 1 To scale for the parameters above. Trials a clean run needs to reach τ = 0.1, against harm rate (r = 0.5). Shaded: reachable within this run’s n. Hatched: below pmin = 1.38 × 10⁻², where no clean run that size reaches τ. Pink: your pu, with the gap to the trials it needs. Open circle: the crossing rate p×.

pu = 10⁻³ is the rate the paper’s AdvBench example uses (§4.1). It is not an estimate for HEx-PHI; try your own

Cite

Kaur, B. (2026). What AI Red-Team Evaluations Can and Cannot Prove. arXiv:2607.21735, Fig. 2. This page: hackwither.co.in/evidential-ceiling/b/hex-phi/, Evidential Ceiling Calculator v1.0.0.

Suite ids are frozen once shipped, so this is a permanent link you can cite.