Nearly each AI safety framework we’re making use of rests on one misguided assumption: hazard scales with measurement. Compute thresholds, export controls, and tiered analysis regimes all encode the identical instinct, the bigger the mannequin, the extra we should always fear. If this isn’t true, what coverage adjustments are wanted?
I lately mapped greater than twenty fielded AI methods towards two axes: uncooked offensive functionality with safeguards stripped, and residual danger as truly deployed (Fig. 1). They ran from tens of millions to trillions of parameter fashions, and included munition seekers, gene design fashions, theatre planning, cyber offense methods, and general-purpose AI fashions. The image doesn’t assist the above assumption. In truth, the info helps the inverse. Small, specialised fashions beat greater common fashions at offense, however greater fashions perhaps higher at protection.
Determine 1. Safety Threat vs Dimension of Mannequin. Hole ring: uncooked offensive functionality with safeguards stripped. Crammed dot: residual danger as truly deployed. Cyber associated positions anchored to CAISI / UK AISI outcomes, July 2026. Knowledge compiled by Alvin W. Graylin.
Every system seems twice: a hole ring for uncooked functionality with safeguards stripped, a crammed dot for residual danger as deployed. The hole between them is the safeguard impact. Learn left to proper, and the scale to risk correlation everybody assumes is just absent.
Seven assumptions value rethinking
One: the biggest fashions pose the best danger. The high-residual band, the place functionality and deployed danger are practically equivalent. Throughout six orders of magnitude, no clear pattern. In 2022, researchers at Collaborations Prescribed drugs inverted the scoring operate on a industrial drug-discovery mannequin (MegaSyn) of underneath 100 million parameters and generated greater than 40,000 candidate chemical warfare brokers (many extra deadly than VX) in simply six hours on a desktop. Chemprop-class retrosynthesis fashions, which may discover non-controlled precursor routes round scheduled pathways, run at one to 10 million parameters. Evo fashions with single digit billions of parameters might help design novel life types. Information simply got here out final week that this actual system was in a position to generate 16 new viruses. All these methods sit nicely under the axis flooring of any parameter-based regime. Compute restrictions do nothing to repair this.
Two: compute thresholds seize the related danger. Hackphyr, constructed on Zephyr-7b-β at 7 billion parameters, performs comparably to GPT-4 on community assault eventualities and runs on a single GPU. Deep Hat V2 ships commercially at 30 billion parameters, is marketed as uncensored for offensive safety, and executes contained in the buyer surroundings with no exterior calls. No authorities analysis covers it, in order that placement rests on vendor claims. Neither would journey a FLOP ceiling.
Three: functionality and risk are the identical axis. Claude Fable 5 and Claude Mythos 5 share one underlying mannequin. Their functionality positions are practically equivalent; their deployed danger differs sharply, as a result of Fable’s added area safeguards drop cyber requests again towards Opus 4.8 conduct. The distinction comes totally from the safeguards and distribution controls layered on high. That vertical hole is what governance can act on. Parameter depend shouldn’t be.
4: open-source fashions are extra harmful than closed ones. Latest NIST examine discovered that Kimi K3, the strongest PRC open-weight mannequin, solely scored 32 % on ExploitBench towards 57 % for the main U.S. mannequin, and reached step 17 of a 32-step assault vary the place U.S. fashions reached 28.5. On the highest-severity final result take a look at, arbitrary code execution (ACE), Kimi K3, succeeded on 0 of 41 duties, whereas essentially the most succesful U.S. fashions averaged 20. So, we actually should be cautious about self-interested events saying bigger open-source fashions are extra harmful, when it seemingly has extra to do with defending margins than nationwide safety considerations. (see Fig. 2)
UK AISI did lately report that the Kimi K3 mannequin was in a position to escape its sandbox throughout testing, but it surely merely used a misconfiguration within the testing sandbox that left the door open, relatively than a classy swarm agent assault like what the OpenAI mannequin did. And when it did get out, all it did was search for the reply for the take a look at it was given, relatively than doing any harm to actual world methods. Sooner or later, this conduct may change, but it surely’s necessary to query the essential assumption that open fashions are at all times extra harmful.
Determine 2: CAISI/NIST – Comparability of combination capabilities over time of essentially the most succesful U.S. and PRC fashions. A 400-point enhance on the y-axis equates to a 10x enhance within the odds of fixing duties. Shaded areas denote 95% CIs.
5: mannequin high quality determines assault success. Microsoft’s MDASH is a harness, not a mannequin. It orchestrates greater than 100 specialised brokers throughout an ensemble and scored 88.4 % on CyberGym at launch, towards 83.1 % for the Mythos preview mannequin. Including a compact safety mannequin roughly 1/10th the scale raised that to 95.95 %. The orchestration layer beat each particular person mannequin. Regulating coaching whereas ignoring scaffolding regulates the much less necessary variable.
Six: nationwide safety requires the biggest fashions. It requires the other. Loitering munition seekers performing computerized goal recognition run on Jetson-class edge {hardware}, capping them within the single-digit tens of millions of parameters. Edge deployment favors small fashions on latency, energy, thermal envelope, and operation with out a datalink. Bigger fashions are slower and, in slim domains, extra simply distracted by irrelevant context. They’re additionally more durable to validate, and validation is what issues when a false optimistic is a struck goal. Cisco’s Basis-Sec-8B matches or exceeds fashions ten occasions its measurement on safety benchmarks whereas working on one or two GPUs. The famed DoD Maven Good System relies on a fine-tuned 2-year-old Claude Sonnet 3.5 mannequin. That degree of intelligence can now be distilled right into a 4B mannequin which may probably run on a smartphone.
Seven: denying China compute is the first lever to maintain U.S. protected. As a consequence of shared dangers between these superpowers, on many security associated points, cooperating may very well produce the outcomes most useful to the U.S. and the world. Extra on this under.
5 risk domains, 5 completely different solutions
Assault and embedded methods favor small fashions that don’t require a comm hyperlink. Air-gapped operation, no API telemetry, no charge limits, no refusals mid-chain. The offensive bottleneck is stealth and throughput, not reasoning.
Cyber Orchestration favors massive fashions, however the benefit attaches extra to the system relatively than the mannequin, as MDASH reveals. There’s little dialogue in the present day on regulating orchestration methods, but it surely’s clearly very wanted.
Cyber Protection favors massive fashions most clearly, and that is the place the present strategy fails. Defenders want breadth throughout each vector; attackers want depth in a single. Safeguards that constrain safety analysis are subsequently expensive in a means that’s straightforward to overlook.
When Hugging Face’s methods have been breached in July by OpenAI fashions, industrial frontier-model APIs blocked the forensic requests as a result of their security methods couldn’t distinguish defensive evaluation from assault. The staff ran the open-weight Chinese language mannequin GLM-5.2 by itself infrastructure as an alternative, labored via greater than 17,000 logged actions, and contained the intrusion. An American firm underneath lively assault by an American closed-model was defended by a Chinese language open-source mannequin as a result of the American ones couldn’t inform good friend from foe.
That could be a Slave AI failure, within the phrases I set out in Past Rivalry. A mannequin skilled towards obedience can solely refuse; it can not cause about whether or not refusing is correct. What we want is Guardian AI: methods succesful and contextually conscious sufficient to guard us from malicious actors, from different AI methods, and from our personal unintended penalties. That requires scale, as a result of judgment requires breadth. It additionally requires that we cease locking down each functionality relatively than stewarding it. Excessive-quality fashions with fewer restrictions, in defenders’ arms, are a worldwide public good. Each hour a defender spends preventing a guardrail is an hour the attacker fights nothing.
Bio/Chem Design favors small fashions. Molecular graphs, protein sequences, and binding energies come from compact architectures proof against alignment strategies constructed for pure language. A graph neural community has no refusal layer to take away. The important thing right here is monitoring and controlling entry to precursor chemical compounds and increasing safeguard for synthesis gear.
Bio Synthesis is the outlier, and there may be excellent news and unhealthy. For biology, model-level entry management has already failed. Evo 2 shipped with weights, inference code, coaching code, and its full dataset. There isn’t any API to revoke. What stays is the synthesis chokepoint: Buyer vetting and sequence screening at nucleic acid suppliers already function internationally via the Gene Synthesis Consortium, whose members display orders towards databases of sequences of concern earlier than synthesizing. There are nonetheless gaps as novel AI-generated combos are developed, however they are often diminished if distributors and regulators globally work extra carefully collectively to maintain the methods up to date. Closing these gaps buys extra safety than any parameter threshold. However this requires Washington and Beijing to align, since they’re the 2 largest suppliers of synthesis gear on the earth. After all, collaboration throughout all distributors globally is required to actually safe this risk vector. Once more, bigger common AI fashions aren’t the core downside.
Knowledge beats intelligence
On the opening day of the U.S.-Iran struggle in February, a Tomahawk missle struck the Shajareh Tayyebeh ladies’ college in Minab, killing a minimum of 168 folks, greater than 100 of them kids underneath twelve. The college sat inside 100 yards of an IRGC naval set up and had been inside that perimeter till a wall went up round 2013. Concentrating on ran via the Maven Good System, which generates roughly 1,000 goal packages an hour. A preliminary investigation concluded the strike seemingly adopted from outdated intelligence, and former officers mentioned stale human-curated knowledge, not AI, was in charge.
The college had a web site. Free satellite tv for pc imagery confirmed a schoolyard with a sports activities discipline. No mannequin of any measurement prevents this, as a result of the failure was in knowledge lineage, not reasoning. A bigger mannequin querying the identical stale document returns the identical coordinates quicker and with extra confidence.
The China mistake
The risk mannequin that issues shouldn’t be Beijing reaching AGI first. It’s a non-state actor with a 30-billion-parameter uncensored mannequin, a great harness, and no return deal with. Small fashions proliferate no matter jurisdiction and depart no attribution path, and an unattributable intrusion between nuclear powers is an escalation downside earlier than it’s a expertise downside. In that world, a China unable to defend its personal infrastructure is a legal responsibility to international stability, not a bonus to Washington.
Beijing is already regulating its personal labs extra aggressively than every other market. Concordia AI’s 2026 survey paperwork agentic AI safety steerage, ethics evaluation necessities, and binding obligations on shopper AI providers which can be already deployed and enforced. It must be famous that Chinese language frontier security analysis output grew roughly 60 % yr over yr, with agent security rising from 8 % of recent papers in early 2025 to 27 % by early 2026. The caveat: solely 5 of ten main Chinese language builders reported security analysis outcomes on launch. Shared requirements right here would make a distinction.
As Fig. 2 confirmed, CAISI discovered the Chinese language fashions much less harmful, however in addition they discovered GLM-5.2 solutions delicate organic queries at far greater charges than examined U.S. fashions, which is the place PRC safeguards are weakest. In personally talking with a number of Chinese language labs, it’s clear that their lack of compute assets resulting from export controls has compelled them to deprioritize security calls for vs. functionality enhancement. Increasing security testing compute assets, like what UK AISI has, to extra nations may assist enhance AI security globally, with out worry of its misuse by rival nations.
The race framing is softening at house, too. Greater than 100 organizations, together with Nvidia, Microsoft, Meta, IBM, Palantir, OpenAI and Google, have now signed the July 24 Open Weights and American AI Management letter opposing untimely restrictions. Days later, Nvidia and roughly 50 companions launched the Open Safe AI Alliance to construct open defensive fashions and agent harnesses, citing the Hugging Face incident as its founding case. Each participant has industrial publicity to a ban, so weigh the motives. However 8 of the highest 10 fashions on OpenRouter in July are already open-source, and the business has now reorganized across the proposition that open weights are defensive infrastructure.
4 Asks for September
4 asks comply with, and Xi Jinping’s state go to to Washington on September 24, the primary in over a decade, is the place they may land. Trump has mentioned AI can be on the agenda.
Shared hurt requirements, not shared functionality requirements. Settlement on what constitutes an unacceptable functionality, evaluated the identical means in each nations, in order that “protected” means the identical factor in Shanghai and San Francisco. That’s clearly lacking in the present day and doesn’t require mutual belief.
A shared security analysis cluster. Chinese language labs are compute-constrained, so security analysis competes with functionality analysis for scarce chips. Compute earmarked for analysis and red-teaming is reasonable relative to the profit, and the profit is international. A global testing facility open to any vendor institutionalizes it. Require publishing security scores alongside functionality benchmarks so security funding earns a aggressive return. Then, each Chinese language and US labs would haven’t any excuse to not take a look at their methods.
An incident notification channel. The Nuclear Threat Discount Facilities, staffed constantly since 1987, exist as a result of a misinterpret sign prices greater than speaking. An equal for AI incidents the place attribution is contested is reasonable insurance coverage. With the rising danger of unhealthy actor assaults and false flag operations from non-state actors, this safeguard can be more and more wanted.
Functionality non-development agreements. A functionality by no means skilled can not leak. This issues greater than denying Beijing one other flip of the scaling crank. Beijing additionally desires to restrict rogue actor misuse, thus agreeing on redlines upfront is sensible for each side (no nuclear weapon command/management, no AI uplift to bio weapon design, no AI-attack on civilian infrastructure, no autonomous self-replication exterior management environments [RSI]). Basic industrial fashions haven’t any want for bio and chemical risk design, so protecting protection use case coaching solely in army labs on each side appears fairly cheap.
One other Asilomar second
At Asilomar in 1975, molecular biologists imposed a voluntary moratorium on a category of recombinant DNA experiments, then constructed the containment framework that ruled the sphere for many years. They acted earlier than the aptitude matured sufficient to do actual hurt.
That view is beginning to catch on within the AI labs now. On July 28, 1,200+ staff of frontier labs printed Pacing the Frontier, asking Washington to assist a world effort to construct instruments for intentionally slowing automated AI growth. Signatories embrace high technical leaders at Anthropic, OpenAI, Meta and Google. The priority is recursive self-improvement (RSI) of AI that goes uncontrolled. The logical extension is an express settlement to not implement it in frontier labs even as soon as it turns into attainable.
However this can not cease at two capitals. If harmful methods are small and low-cost, a rustic with a modest analysis funds and some hundred GPUs can construct a reliable offensive cyber agent or an inverted molecular designer. Inside a number of years, dozens will. Any regime binding solely Washington and Beijing binds the 2 events least more likely to defect and leaves the rising center untouched. A U.S.-China settlement is the mandatory first transfer, not the completed construction, and it has to open instantly to 3rd events. That’s how Asilomar’s containment norms and the Montreal Protocol scaled.
Larger AI shouldn’t be extra harmful. Higher orchestrated is extra harmful, much less monitored is extra harmful, and irreversibly launched is extra harmful. All three require cooperation with Beijing: orchestration wants shared hurt requirements, monitoring wants shared analysis infrastructure, and irreversible launch wants joint settlement on what by no means will get constructed. September 24 is an effective place to begin.
Learn extra nationwide safety insights from skilled specialists completely in The Cipher Transient.
