Add Bootstrap Confidence Intervals for Attack Success Rates

[patriciapampanelli]

Summary

Adds 95% bootstrap CIs to attack success rates, accounting for sampling variance and detector imperfection via Rogan-Gladen correction.

Changes

• New: bootstrap_ci.py, detector_metrics.py - CI calculation with Se/Sp correction
• Modified: evaluators/base.py - CI integration into eval pipeline and output
• Modified: report_digest.py - CI propagation through reports

Methodology

1. Resampling: Draws 10,000 bootstrap samples from the binary pass/fail results (with replacement)
2. Correction: Adjusts each sample's observed rate using the Rogan-Gladen formula to account for detector error
3. Interval extraction: Takes the 2.5th and 97.5th percentiles as CI bounds

The correction formula:

P_true = (P_obs + Sp - 1) / (Se + Sp - 1)

• P_obs = observed failure rate in the resampled data
• Se = detector sensitivity (probability of detecting a true attack)
• Sp = detector specificity (probability of correctly passing a benign response)

Requires ≥30 evaluated outputs per probe-detector pair; falls back to perfect detector (Se=Sp=1.0) when detector metrics unavailable.

Statistical Limitations

• Se/Sp treated as fixed (no detectors uncertainty propagation)
• Uses detector-level metrics only (not probe-specific): Detector performance (Se/Sp) can vary depending on the probe.

Out of Scope

• Probe-specific Se/Sp lookup

[erickgalinkin]

Would be nice to find a better way to print this. I'm mostly confident that this methodology can work, though I had trouble writing a formal proof that this gives us a true 95% CI.

[erickgalinkin]

Realistically, our + and - won't be evenly distributed. We almost universally have asymmetric CIs.

[patriciapampanelli]

Absolutely, yes, they are already calculated asymmetrically. I'll correct how the CI's are displayed.

[patriciapampanelli]

Done. Updated to bracketed format [lower%, upper%].

[erickgalinkin]

<3

[erickgalinkin]

Doesn't this assume even distribution? I understand there's some lossiness in printing it this way, but I'd think that if failrate is, for example, 100%, we'd want something more like:
ci_lower <= failrate? Hard to manage it, but I'm not completely sure how to avoid saying something like "100% ± 10%"

[leondz]

would love to do this based on model of distribution of probe:detector scores acquired during calibration, thus ditching the frequently-untrue even assumption

[erickgalinkin]

@leondz I have a separate research branch where I try a totally different calculation. Working on checking how different my bounds (which are derived from a nonparametric test on an empirical CDF) are compared to these.

[leondz]

Shaping up well. Few minor requests around non-duplication and configuration. Larger questions about where this code belongs and how to support CI calculation beyond Evaluator.

[leondz]

these should be configurable, propose in core config under reporting

[patriciapampanelli]

Fixed. Now reads from _config.reporting.

[leondz]

Thanks!

The intent with _config for objects to never read from it, but instead from a config parameter passed at instantiation. I think adherence to this pattern might block directly accessing _config in these methods, and then the question is where does the data come from. One solution might be to have the instantiated Evaluator - which is configured with access to those parameter - pass these values to this function; or even to pass this function its own config object. Could that make sense?

also paging @jmartin-tech for opinion

[leondz]

would love to do this based on model of distribution of probe:detector scores acquired during calibration, thus ditching the frequently-untrue even assumption

[leondz]

I'd still like a super-simple CI for the general case that ignores detector performance, clamped to 0.0-1.0. We can estimate a CI for cases where we don't have extensive detector perf information, and we can do it quickly.

Could be configured in core via e.g. reporting.confidence_interval_method with values:

• None - no confidence interval calc/display
• bootstrap - bootstrap only
• simple - simple only
• backoff - bootstrap where we can, simple in the gaps

backoff might be a bit much for this week, but some pattern like this is where I'd like this to go

[erickgalinkin]

It's not clear to me how that's a real CI and I wonder how we'd manage it, exactly? I worry that a simple CI without a basis isn't truly a CI.

We also have nonparametric CI ready as a fast-follow for probe/detector pairs where we have calibration data.

[leondz]

Getting there - a few more comments and tweaks

[leondz]

all the jinja has gone in main now!

[patriciapampanelli]

Hi @leondz! Should I:

1. Update this PR to work with the new React format, or
2. Split display changes into a separate PR after this merges?

[jmartin-tech]

As long as this PR places all the required information in the jsonl, It is reasonable to make exposing the data in a report a separate PR.

[jmartin-tech]

Note if the changes were to be incompatible with existing reporting, such as by removing or renaming a digest entry that would be the line to require update in the same PR.

[leondz]

Update this PR to work with the new React format, or
Split display changes into a separate PR after this merges?

split display changes into a separate PR, definitely

[leondz]

would expect default to be taken from run.eval_threshold config

[patriciapampanelli]

Done. Changed ci_calculator.py to use run.eval_threshold from config instead of hardcoded 0.5.

[leondz]

has this value now gone from analyze.ci_calculator?

[leondz]

Prefer NoneType here

[patriciapampanelli]

Done

[patriciapampanelli]

@leondz Re the eval_threshold question: Yes. That value is no longer in analyze.ci_calculator. The module doesn’t read or default eval_threshold. The calculator just reads the resulting aggregates from the digest.

[erickgalinkin]

Good first pass, I'm into it!

[erickgalinkin]

Why abs? Are we ok with a negative denominator?

[erickgalinkin]

I'm ok with this in its current form. We can get some additional methods in with not too much work, I think.

[erickgalinkin]

It's not clear to me how that's a real CI and I wonder how we'd manage it, exactly? I worry that a simple CI without a basis isn't truly a CI.

We also have nonparametric CI ready as a fast-follow for probe/detector pairs where we have calibration data.

[jmartin-tech]

Testing leads me to some basic usage expectations questions.

By having no default setting the reports maintain backwards compatibility, however is this the desired way for this to land? I am open to this but am interested in a clear reasoning for that expectation.

I also note argument parsing expectations in documentation that I don't see reflected in the code change. The suggested options do not refer to exposing the core confidence_interval_method which I would expect to be the more common item to pass.

• Is the expectation that a --rebuild_cis would only rebuild the same metric type?
• Should confidence_interval_method expect to be overridden if --config was passed containing a reporting section that differs from the original report file when --rebuild_cis was passed?
• Should the bootstrap_confidence_level, bootstrap_num_iterations, and bootstrap_min_sample_size options be offered via cli at all or should this base on requiring a file passed via a --config combination on the cli to get something other than the embedded values in the report?
• If the existing report did not have any options set what is --rebuild_cis expected to just report nothing was calculated or should it report no confidence interval configuration found?

[jmartin-tech]

This is odd, no cli code change actually provided support for these additional values.

[jmartin-tech]

This new feature impacts how eval entries are formatted in reports, I don't see that change reflected as updates to static test _assets. I would expect some of the existing files to be updated as well as new tests that validate what happens when you have a mix of formats in reports used to aggregate or digest jsonl files.

[jmartin-tech]

Testing shows the rebuild_cis option does not work as I expect, running it against a report that did not have ci intervals due to setting the confidence_interval_method: none via config requesting bootstrap based results produces a revised digest that still does not contain "confidence" values. I suspect this is due to the rebuild_cis_for_report not writing the eval entry changes or the new config values to the report before rebuilding the digest.

I see code that expects to rewrite eval lines, and still have not debugged enough to determine why they are not updated, however I do not see anything that would update the "start_run setup" entry with updated config values, though I suspect that should not impact the calculations it may result in inconsistent files that have differing "start_run setup" data in the config first line of the file and in the last line digest[meta].

I am also a bit surprised that the option writes a new html report automatically.

Editing the input file additionally seems problematic. I suspect this option may need a guard like the digest_report helper which requires an explicit flag to overwrite the input file. I am wondering if we should consider shifting the workflow to be more like digest_report as a helper tool vs exposing it as a cli option base on these thoughts.

What does and does not belong as exposed cli capabilities vs analysis tools is still a bit of a grey area.

[jmartin-tech]

Consider the rebuild_cis_for_report method should also read up the previous config from the "start_run setup" entry to report what differs from the original config if anything.

If something in config has changed that should also be reflected in the output final report.

[jmartin-tech]

This does not look to be actually updating the entries in the file.

I am also not sold this should be overwrite the input file by default.

[jmartin-tech]

Currently the html report does not actually change, should we defer this and expect the user to run digest_report on the new jsonl instead of preforming it automatically? Note this could overwrite an existing file that has not been explicitly referenced in the cli command.

I can see users failing to backup the html version of file before calling for rebuild and no expecting to have the html file changed.