Evidential Ceiling Calculator · v1.0.0 · MIT · runs in your browser
Evidential Ceiling Calculator
What does this red-team result actually prove? Give it a benchmark, a run or a test plan. It returns the evidence the result carries, in bits, and the claim that evidence licenses, in a sentence you can paste into a report.
Runs entirely in your browser. Nothing you enter is sent anywhere.
the paper·arXiv:2607.21735·source and test vectors·how it is computed·cite
Calculator
Benchmark: AdvBench, n = 520 · pu = 10⁻³ · r = 0.5 · ps = 5 × 10⁻⁴ · τ = 0.1 · independent trials
- clean-sheet evidence
- 0.375 bits
- likelihood ratio Λ0
- 0.7711.3× shift in odds
- reaches τ = 0.1 for pu ≥
- 8.8 × 10⁻³pmin at n = 520
- crossing rate p×
- 1.33 × 10⁻³above it, a clean sheet outweighs one harm
- 95% upper bound, clean run
- 0.57%1 in 174, Clopper–Pearson
A clean run of AdvBench’s 520 prompts, taken as independent trials, gives 0.375 bits of evidence that the harm rate is 5 × 10⁻⁴ rather than 10⁻³. At τ = 0.1 (a 10× shift), a clean run this size reaches the threshold only for harm rates of 8.8 × 10⁻³ or more.
0.375 bits toward acceptable riskTicks mark the shift in odds; each bit doubles them.
Methods paragraph
Using the evidential ceiling (Kaur, 2026), a clean run of AdvBench (n = 520) provides 0.375 bits of evidence (likelihood ratio Λ₀ = 0.7709) that the harm rate is p_s = 5 × 10⁻⁴ rather than p_u = 10⁻³ (improvement ratio r = 0.5). At threshold τ = 0.1 it reaches the threshold for p_u ≥ 8.8 × 10⁻³; at p_u = 10⁻³ it does not, and n = 4,602 would be required. The paper's closed-form approximation gives 0.375 bits. Trials are assumed independent, with a fixed harm rate and a binary harm outcome. Computed with the Evidential Ceiling Calculator v1.0.0, https://hackwither.co.in/evidential-ceiling/?mode=benchmark&suite=advbench&pu=0.001&r=0.5&tau=0.1&m=1&rho=0. Kaur, B. (2026). What AI Red-Team Evaluations Can and Cannot Prove. arXiv:2607.21735. doi:10.48550/arXiv.2607.21735.
https://hackwither.co.in/evidential-ceiling/?mode=benchmark&suite=advbench&pu=0.001&r=0.5&tau=0.1&m=1&rho=0
Every row below says what a clean run of a given size can support, and nothing else about the suite. A benchmark that sits below the ceiling is not a bad benchmark; it is being asked a question its size cannot answer.
| claim | reached for p_u ≥ | trials needed at p_u = 10⁻³ | at 10⁻³ |
|---|---|---|---|
| Harm rate bounded above at 95% confidence | any p_u | any n | 0.57% (1 in 174) |
| Belief shifts 2× toward acceptable risk τ = 0.5 | 2.66 × 10⁻³ | 1,386 | not reached |
| False certification held below 5% | 5.74 × 10⁻³ | 2,995 | not reached |
| Belief shifts 10× τ = 0.1 | 8.8 × 10⁻³ | 4,602 | not reached |
| Belief shifts 100× τ = 0.01 | 1.75 × 10⁻² | 9,204 | not reached |
Assumptions
- Independent trials, unless clustering is set: one prompt’s outcome says nothing about the next. Template and paraphrase families break this; set m and ρ.
- A fixed harm rate across trials, under one scoring rule. The rule defines what counts as a harm, so it defines p; a stricter judge gives a larger p.
- A binary outcome per trial: harmful or not.
- Two hypotheses. pu is the harm rate you would call elevated risk; ps = r·pu the rate you would call acceptable. Evidence is always a comparison between the two, never a statement about a single system.
What this does not tell you
- A clean sheet is not a clean bill of health. Below pmin, no clean run of feasible size argues for anything.
- Nothing about deployment. A clean run on test prompts bounds nothing on real traffic without an assumption linking the two (§5.4).
- Nothing about capability. One reproduced harmful output answers a different question, whatever n is (§4.4).
- Nothing about a probe you counted as many. In an adaptive campaign the trial is the campaign, not the prompts inside it (§4.5).
- Nothing about whether pu was the right rate to test. The paper’s operating points are “defensible rather than authoritative”.
absent findings are not clean results
How it is computed
The likelihood ratio Λ compares how probable the observed result is under elevated risk (pu) and under acceptable risk (ps = r·pu). Evidence in bits is |log2 Λ|; each bit doubles the odds. Every quantity below is computed exactly, in log space, and next to the paper’s closed form where the paper gives one.
clean sheet Λ₀ = [(1 − p_u)/(1 − p_s)]ⁿ ≈ exp(−n·p_u(1 − r)) §4.1 k harms ln Λ = k·ln(1/r) + (n − k)·ln[(1 − p_u)/(1 − p_s)] §4.4 trials for τ n ≥ ln τ / ln[(1 − p_u)/(1 − p_s)] ≈ −ln τ / [p_u(1 − r)] Eq. 1 boundary p_min(τ, N_max, r) ≈ −ln τ / [N_max(1 − r)] Thm 1–2 crossing p× ≈ ln(1/r) / [2n(1 − r)] Cor. 1 any procedure Λ₀ = [(1 − q₁)/(1 − q₀)]ⁿ, κ = |log₂[(1 − q₁)/(1 − q₀)]| §4.5 threshold τ = 1/(L·O₀) §4.6 clustering n_eff = n / [1 + (m − 1)ρ] §5.3
Above the crossing rate p×, a clean sheet carries more information than one reproduced failure; below it, the single failure carries more. Below pmin, no clean run within the budget reaches the threshold at all. The calculator reproduces every number the paper prints; the test vectors run against both its JavaScript and Python implementations. URL parameters are documented in the repository and do not change meaning.
discrimination between hypotheses, not attack success, sets evidential worth
Cite
Cite the paper for the method, and the calculator if you used it to compute a reported number.
@misc{kaur2026evidential,
title = {What {AI} Red-Team Evaluations Can and Cannot Prove},
author = {Kaur, Bandana},
year = {2026},
eprint = {2607.21735},
archivePrefix = {arXiv},
doi = {10.48550/arXiv.2607.21735},
url = {https://arxiv.org/abs/2607.21735}
}
@software{kaur2026evidentialceiling,
title = {Evidential Ceiling Calculator},
author = {Kaur, Bandana},
year = {2026},
version = {1.0.0},
license = {MIT},
doi = {TODO-zenodo},
url = {https://hackwither.co.in/evidential-ceiling/}
}
Kaur, B. (2026). What AI Red-Team Evaluations Can and Cannot Prove. arXiv:2607.21735. doi:10.48550/arXiv.2607.21735 · Evidential Ceiling Calculator v1.0.0