Europe is turning the promise of AI safety evaluation into a procurement and measurement problem. A newly announced cyber-risk consortium could help regulators ask harder questions of advanced models. But its success will depend less on whether it can produce a benchmark score than on whether it can demonstrate what a model can do through tools, under realistic constraints, and after repeated attempts.
On November 5, 2026, Capgemini said that the European Commission’s AI Office had selected a consortium it leads for cyber-threat modelling and assessments of advanced general-purpose AI models. Partner Wavestone confirmed its participation on November 6; France’s testing institution LNE is also named. ITPro separately reported the announcement. These accounts corroborate the public announcement, but they are not a substitute for an independently inspected Commission award notice, which we have not verified. The consortium describes planned work on evaluation tools, technical methods, cyber-risk modelling and on-demand assessments.
The November announcement reports a selection, not a completed model audit, an independent evaluation of a named model or evidence that any model is unsafe. The European Commission’s earlier procurement material establishes the existence and scope of the tender, not the identity of its eventual winner. Whether the contractual award and scope match every detail of the consortium's publicity remains a question for an official award record. Only subsequent methods, results and oversight can establish whether the work produces reliable conclusions.
The regulatory demand precedes the test method
The EU AI Act’s Article 55 requires providers of general-purpose AI models classified as presenting systemic risk to evaluate them with state-of-the-art, standardised protocols, including documented adversarial testing; assess and mitigate systemic risks; report serious incidents; and maintain cybersecurity protections. The question is how those obligations become operationally meaningful when model capabilities, tools and threat techniques change faster than conventional testing programmes.
The chronology matters. The Commission first described a technical-support tender in July 2025, before the consortium's November 2026 announcement. That earlier programme envisaged separate risk areas, including cyber offence, CBRN, loss of control, manipulation and sociotechnical risks, as well as evaluation of agents in dynamic settings. A July 2026 external-evaluator workshop, documented in the Commission's October 2026 report, addressed qualifications for independent assessments. These milestones describe related regulatory preparation; none independently proves the subsequent consortium award.
This arrangement recognises a practical challenge: a single headline score cannot capture the spread from a chatbot answering security questions to a deployed agent with a terminal, browser, credentials and time to iterate. Evaluators must describe which configuration they tested and which capabilities the configuration did or did not expose.
A model’s exam result is not the same as a successful attack
A conventional benchmark might ask an AI system to identify a vulnerability or explain a code sample. That can test knowledge. A more realistic exercise asks whether the model, in a controlled environment, can navigate a sequence of decisions: understand a target, use permitted tools, interpret errors, choose a new approach and complete an objective within a time and compute budget. The measured thing is different, and so is the risk.
The UK’s AI Security Institute measures how long a human expert would take to complete cyber tasks on which models reach an estimated 80% success rate. Its narrow test suite limits runs to 2.5 million tokens per task and excludes many conditions of real attacks. AISI explicitly identifies uncertainty: only six of its suite's tasks last eight hours or more, some human baselines are expert estimates rather than timed trials, and newer models approach the suite's measurement ceiling. It says the recent faster trend may not persist. These are documented limitations of AISI's benchmark, not findings about the EU consortium's future audits.
This is an important methodological caution for any regulator. A system that fails a multi-hour simulated task might succeed if paired with better software tools or a human operator. A model that passes a set of challenges might be much less effective against defended networks with unfamiliar environments, active monitoring and unpredictable side effects. A serious audit reports both directions of uncertainty. This evidence boundary matters beyond cyber benchmarks; our reporting on a METR monitoring pilot likewise distinguishes historical evaluations from claims about later fixes.
Five measurements that should accompany every capability number
First, the test environment. Evaluators should document whether the model has shell access, browsing, internet connectivity, code execution, persistent memory and external credentials. An agent denied external connections is being asked a different question from one that can perform authenticated actions against a live service. Reporting only the base model’s name omits critical context.
Second, the budget. Number of attempts, elapsed time, inference tokens and available computational resources all affect what a system can achieve. If an agent succeeds after repeated trials, a single-run score hides a material part of the outcome. If a benchmark cuts off work before useful progress ends, it may understate capability.
Third, the task distribution. Real incidents involve many paths, from reconnaissance through exploitation and persistence, while benchmarks typically choose a narrower set of tasks that are safe and repeatable. Passing a challenge may reflect familiarity with that challenge rather than transferable operational skill. Evaluators need holdout tasks, periodically refreshed datasets and checks for accidental exposure of benchmark content during training.
Fourth, the defensive environment. The same agent may behave differently against a sandbox with no monitoring than against a production-like setting with rate limits, access controls, logging and active response. Evaluation should distinguish raw offensive ability from the ability to achieve objectives despite plausible defences. A risk estimate without that distinction can mislead both defenders and policymakers.
Fifth, statistical uncertainty and reproducibility. A few successful attempts do not establish a robust success probability. AISI publishes uncertainty bands and explains that its narrow task suite covers only some of the capabilities relevant to real attacks. Such estimates require repeated trials, uncertainty intervals and clear coverage limits before they can be used in policy. This is a proposed reporting standard for future EU assessments, not a claim that the consortium has already adopted or failed it.
Independence is an operational property, not a label
There is also a governance problem: the provider knows the model best, while an external auditor may lack enough access to test it properly. The AI Office explored qualifications and independence requirements for outside evaluators in a July 2026 workshop and published an event report in October. Independence should not be assumed merely because an evaluation company is formally separate from a model developer.
An evaluator may depend on a provider for access to weights, internal tools, safety documentation, model snapshots or reproducible environments. Commercial pressure, confidentiality restrictions and the ability of the provider to select which version gets tested can influence the exercise. The useful questions are concrete: who chooses the tasks, who pays, who can challenge results, who can see negative findings and what happens if a model changes after the assessment?
None of this implies misconduct by Capgemini, Wavestone, LNE or any individual provider. It describes structural risks that every credible evaluation regime must address. These controls matter as much as the sophistication of the red-team techniques.
The counterargument: public benchmarks can teach attackers
There is a principled reason not to disclose everything. Publishing the detailed environment and successful exploit chain for a high-capability agent could make its misuse easier. It might also encourage developers to optimise specifically for the public test while leaving unmeasured behaviour untouched. Some evaluations therefore need restricted reports, confidential evidence and controlled access for regulators and independent reviewers.
But confidentiality does not justify opaque claims of safety. An organisation should still be able to publish the evaluated model version, broad task domains, test budget, statistical reliability, limits of coverage, governance process and a clear statement of whether identified problems were mitigated and retested. A useful audit distinguishes restricted technical detail from publicly understandable accountability.
Testing needs to be adapted to the system and consequences being assessed. Cyber capability measurement is therefore not a one-off certification stamp; it is an evolving engineering discipline. That is our assessment, rather than a finding from a completed EU audit.
What would count as progress over the next year?
Evidence of progress would include published evaluation protocols, validated test environments, documented independent access, uncertainty reporting, a process for models that change between reviews and observable links between findings and corrective measures. Incident-monitoring data should inform the next round of tests instead of remaining in a separate compliance silo. It would also help if the Commission clarified how results from its different risk-area workstreams are combined without double counting or presenting unrelated scores as interchangeable.
For AI vendors, such a regime could become an incentive to improve instrumentation and maintain reproducible release records. For customers, it may make procurement questions more specific: which model build, which tool permissions, which conditions, which audit and which unresolved limitations? For regulators, it offers a route from broad legal language to evidence that can be examined and challenged.
Our assessment: the Commission's published procurement plans and the consortium members' November statements indicate movement toward greater AI evaluation capacity. We have not independently inspected the Commission's award decision or a completed audit. Our argument for end-to-end testing and uncertainty reporting is an editorial recommendation grounded in the methodological limitations documented by AISI; it is not a stated contractual requirement of the consortium. The useful outcome would be credible evidence about the boundary between measured capability, deployment conditions and real-world cyber risk.
Illustration: original AI-generated conceptual diagram of an AI evaluation grid and system boundary. It does not depict a completed EU audit.
Sources and scope
- Capgemini, November 5, 2026: Consortium's own selection announcement (interested-party account)
- Wavestone, November 6, 2026: Consortium partner confirmation
- ITPro: Reporting on the announced consortium
- European Commission, July 10, 2025: GPAI safety technical-support procurement
- EU AI Act Service Desk: Article 55 statutory obligations
- European Commission, October 21, 2026: External evaluator workshop report
- UK AI Security Institute: Cyber-task duration evaluation methods and limitations