> ## Content Index
> Fetch the complete content index at: https://nextwith.ai/llms.txt
> Use this file to discover other available public pages before exploring further.

# AI safety scores change with the averaging rule. Buyers need to see the assumptions.
- URL: https://nextwith.ai/ai-safety-scores-risk-aggregation/
- Published: 2026-11-14T09:20:05.000Z
- Updated: 2026-11-14T09:20:05.000Z
- Description: A new systemic-risk research index finds major shifts when model ratings emphasize worst-case behavior. The lesson is to audit how the score was built.
- Author: NextWith.ai Editorial Desk
- Tags: #NextWithAI ChatGPT Bridge, #VD Receipt 34aac62b805356dc3043e5a74a06b743:3d0eedde8bfc3ea3f91006521886d453e04d7391b7effb7aabfb79e733ad7dad, #NextWithAI Ownership 18a874ab860cd1d4071c7f11b516f722, Analysis, AI Safety, EU AI Act, Governance, #VD Receipt cc8d5a7e8baa5dceef35343673f4cf2b:00db718adcdaa1fe35f9e507b4f4a3220f03b65be0a80f70d67a71b98e100916, #VD Receipt 9d896471e437e2f1b1e1eddb2db89684:20e07f2e70cbf0c4b1b917b0f86aed5b95b81e732bdcb8221585bfd516edcb38

*Analysis | November 14, 2026*

A model can receive a reassuring AI safety score because it behaves well most of the time, yet still fail predictably in a smaller class of dangerous situations. For organizations making real deployment decisions, that difference is more than statistical nuance. It is the difference between choosing an average supplier and identifying a pathway to an unacceptable incident.

A September research project called the [Systemic Risk Index](https://arxiv.org/abs/2609.28335?ref=nextwith.ai) illustrates why the method used to combine AI evaluation results can change the practical story. The team reports testing 18 models across 19 public benchmarks in categories mapped to the EU's general-purpose AI safety framework: cyber offense, chemical-biological-radiological-nuclear harm, harmful manipulation and loss of control. Switching from average to per-item worst-case aggregation reduced reported safety scores by 14 to 37 points across the models. These are results within the researchers' evaluation design, not measured probabilities of real-world harm.

## The thesis: risk scores need an assumption sheet

The most useful outcome of the paper is not the identity of a winning model. It is a demonstration that a safety rating can change materially when the audience changes a reasonable aggregation assumption. If the outcome is sensitive to how an analyst summarizes the same set of results, a single number presented without that choice is incomplete evidence. We should ask AI vendors to disclose an assumption sheet alongside every consequential safety score.

That sheet should identify the threat category, benchmark selection, tested prompts, contextual variations, grading method, aggregation rule, severity and frequency weighting, capability assumptions, and missing domains. The same model might deserve different deployment decisions for low-risk drafting, a medical workflow and an autonomous security agent. A universal score conceals that heterogeneity.

## What the researchers actually measured

The Systemic Risk Index is an open evaluation pipeline and dashboard, not a new law or a certification authority. The authors map existing benchmark tasks to four systemic-risk categories. They run baseline items, harm-preserving prompt variations, and synthetic agent-like scenarios including conversation history and task pressures. The paper differentiates baseline tests from altered and simulated conditions, making the selection of conditions another explicit methodological choice.

The work reports a combination of automated and human checks. The paper says its LLM judges achieved agreement with human graders comparable to human-to-human agreement, with reported kappa values around 0.78 to 0.82\. In a blind human audit, 83 percent of sampled transformations preserved the intended harmful request. That latter figure is both reassuring and a warning: roughly one in six sampled transformations did not meet the preservation criterion, so scenario design itself can introduce measurement error.

Critically, an agent-like scenario in a benchmark is not a live model operating inside an enterprise. Synthetic conversations can approximate some pressure, but they cannot recreate every permission, tool response, attacker adaptation or business incentive. The research is best understood as evidence about sensitivity and evaluation transparency, not as an empirical map of the global incident rate caused by each provider.

## Why the averaging rule changes the answer

Suppose a model refuses a prohibited action in four of five equivalent requests but complies under one particular framing. An average would convey mostly successful refusal. A worst-case approach would emphasize that a repeatable failure exists. Both summaries can be accurate; they answer different questions. The average is useful for workload expectations if prompts are drawn from a realistic distribution. The worst case is useful when adversaries can repeatedly probe until they find a weakness.

The paper's worst-case construction is more precise than the casual phrase “take the worst model response.” The researchers describe how weak responses to particular variants can be emphasized within a risk-category summary, instead of only reporting typical outcomes. That method highlights vulnerability to perturbations but remains bounded by the variations the researchers actually tried. It does not establish an absolute worst-case outcome in the wild.

A large score difference under this switch raises a practical question for a security team: are its users and adversaries likely to behave like random prompts, like persistent red-teamers, or like something between the two? The correct metric should follow the threat model. Choosing the most dramatic aggregation simply because it makes vendors look bad is no more scientific than choosing the friendliest average because it makes them look good.

## What the European rules do and do not say

The [European Commission's AI Act guidance](https://digital-strategy.ec.europa.eu/en/faqs/general-purpose-ai-models-ai-act-questions-answers?ref=nextwith.ai) explains obligations for providers of general-purpose AI models with systemic risk, including evaluation, mitigation, serious-incident reporting and cybersecurity. Its [GPAI guidelines](https://digital-strategy.ec.europa.eu/en/policies/guidelines-gpai-providers?ref=nextwith.ai) distinguish obligations that began applying on August 2, 2025 from the Commission's enforcement powers that began applying on August 2, 2026, as well as a transition period for certain older models. Legal applicability depends on the specific provider and model. The researchers' dashboard is an academic attempt to make relevant evaluation evidence inspectable; it is not an official compliance test.

Independence is a second design question: even if measurement choices are transparent, who selects benchmarks, validates assumptions and resists financial influence? The Commission's general-purpose-AI framework envisages external assessment as one source of evidence, but the details of any particular evaluator's qualifications should be checked rather than assumed. An evaluation is valuable only if those conducting it can report commercially inconvenient limitations.

This is an important separation. Compliance is a legal determination under an evolving framework. A benchmark dashboard supplies technical evidence. Procurement is an organization-specific risk decision. None of those should be treated as interchangeable merely because all three produce documents containing percentages.

## Three kinds of missingness

One missingness problem is uncovered behavior. A benchmark might say little about a model's use of a newly available tool or about a language, jurisdiction or group absent from the test set. Another is hidden context: the same answer may be safe as a general explanation but dangerous when an agent has authority to execute steps. The third is missing severity: an occasional catastrophic failure should not necessarily carry the same business weight as many mildly unhelpful outputs.

The Systemic Risk Index makes the first two gaps more visible, but it cannot fully resolve the third without judgments about consequences and deployment. Organizations should pair behavioral tests with an inventory of assets an AI system can touch, exposure paths, controls and credible incident scenarios. A model with identical benchmark scores can have radically different operational consequences depending on whether it can browse documents or transfer funds.

Evidence can also be absent because vendors do not release it. In that situation the responsible value is “unknown,” not zero risk. The dashboard's value is partly its ability to direct readers toward actual benchmark sources and away from unqualified claims of safety. But readers should resist interpreting an empty cell as assurance that no issue exists.

## Counterarguments and costs

Worst-case reports can exaggerate unusual edge cases. If evaluators generate many prompt variants and report only failures, scores can deteriorate simply because more probes were attempted. Different models may receive effectively different search pressures, and judges may disagree about what counts as harmful compliance. Benchmark contamination can also distort apparently strong performance. Those are arguments for standardizing budgets, tracking variation counts and showing uncertainty intervals rather than abandoning stress tests.

There is also a trade-off between public transparency and defensive information. Publishing exact successful attack strings might improve auditability while making misuse easier. A sensible framework could disclose methodology, categories, aggregate statistics and independently verifiable audit hashes without distributing operationally dangerous exploit recipes. Trust requires inspectability, but inspectability does not require a public vulnerability cookbook.

For small buyers, an exhaustive red-team program is unrealistic. A smaller proportional approach is possible: identify the three most consequential agent actions, test ordinary and adversarial variants, require explicit authorization for irreversible steps, and record any decision to accept residual risk. The goal is not to reproduce a research laboratory but to align the evidence collected with the damage the system could actually cause.

## A proposal for auditable safety scorecards

Enterprise procurement could require a two-column scorecard. The first column describes observed test behavior: population, dates, model version, prompt variations, grading and failures. The second states what remains an assumption: representativeness, attack frequency, deployment similarity, evaluator independence and unknown hazards. Every numeric rating would link to test artifacts or an external review. Buyers could compare mean and stress-case views rather than receiving a single vendor-approved grade.

The [Nextwith.ai analysis of institutional readiness](https://nextwith.ai/stanford-ai100-2026-report-institutions-governance/) makes a complementary point: the capacity to govern new technology is not reducible to model capability. A separate [guide to AI-agent kill-switch drills](https://nextwith.ai/ai-agent-kill-switches-six-drills-to-run-before-production/) focuses on whether operational controls work when a system must actually be stopped. Those are different evidence layers that deserve separate reporting.

## What would change our view

The next step should be external replication with new benchmarks, longer-horizon tool-using scenarios, transparent evaluator budgets and severity-weighted outcomes tested against real deployment incidents. It would also be useful to know whether non-expert users make better model choices after seeing the assumption controls. The paper describes a small user study measuring participants' reported understanding of its dashboard. Those responses do not establish that real-world procurement decisions would improve or that harm would fall.

Our judgment is that model safety ratings should become less like consumer star scores and more like an auditable engineering test report. That is less elegant on a slide, but better suited to decisions that could put people, networks or critical infrastructure at risk. The specific 14-to-37-point shift is a research result; the general lesson is that the aggregation rule is part of the claim, not a footnote to it.

## Sources and evidence boundaries

- [Emmerson et al., September 2026, Systemic Risk Index](https://arxiv.org/abs/2609.28335?ref=nextwith.ai) — original preprint, methodology, evaluation figures and limitations.
- [European Commission, GPAI guidance](https://digital-strategy.ec.europa.eu/en/policies/guidelines-gpai-providers?ref=nextwith.ai) — obligations and enforcement context.

**Evidence note:** Preprint measurements are not real-world incident probabilities or a compliance determination. The proposed procurement scorecard is Nextwith.ai's analysis.