Things Made Too Dangerous to Release

Every significant capability governance failure of the last fifty years follows the same shape. Someone drew a boundary around an approved use case. The capability moved outside it. Nobody had asked what happens when it does.

That pattern predates AI by decades. It shows up in molecular biology, network infrastructure, industrial control systems, and weapons grade software. Technology changes. The governance failure is the same.

Stuxnet is the most instructive recent example, not because of what it did, but because of what it reveals about the limits of use case governance.

In 2010 a piece of software engineered with surgical precision to destroy centrifuge rotors at a single Iranian facility escaped into the wild. It propagated across thousands of industrial networks worldwide. The destructive payload stayed dormant on non-targeted systems. But the blueprint became public.

The designers had drawn a very precise boundary. The capability ignored it. And the lesson that mattered wasn’t technical, it was that governing the intended use case is not the same as governing the capability itself. Boards approving AI deployments in 2026 are making the same mistake the Stuxnet architects made, at far greater scale and with far less precision about where the boundary sits.

The approval problem

Most organisations govern AI the same way they govern everything else. Someone proposes a use case. Security and legal review it against the intended application. If it falls within acceptable tolerances, it gets approved. The use case is documented. The risk is recorded. The deployment proceeds.

The problem is that approval covers the use case, not the capability.

Those aren’t the same thing. A use case is what you intend to do with a system. A capability is what the system can do. The gap between them is where governance fails, and in most organisations, nobody has mapped it.

A script authorised to automate backend maintenance can be repurposed to delete production databases. A language model deployed to summarise support tickets can be connected to external APIs and given the ability to execute database queries nobody approved. An audience segmentation tool approved as a marketing feature can start making decisions about who gets reached, with what message, under what conditions, without anyone having classified it as a decision making system.

None of those are exotic threat scenarios. They’re the mundane consequence of approving use cases and assuming the capability stays inside them.

The question most approval processes don’t ask is a simple one. What can this system do beyond the use case we’ve approved, and what stops it?

If your current governance process can’t answer that, it’s documenting intentions, not governing capabilities.

The pause is sometimes the control

In 1974 molecular biologists developed the ability to splice genetic material from different organisms to create recombinant DNA. The medical potential was significant. So was the risk engineered pathogens with antibiotic resistance that could replicate and spread outside any laboratory boundary.

The scientific community did something that almost never happens under commercial pressure. They stopped.

Paul Berg and Maxine Singer led a voluntary moratorium on high risk experiments. Not a regulatory pause. Not a government mandate. A self imposed halt by the people with the most to gain from proceeding. They then convened the Asilomar Conference in February 1975, which produced a risk stratified containment framework four tiers, each with mandatory physical and biological containment measures matched to the estimated risk level before any experiment could proceed.

The framework wasn’t perfect. The classification debates were contentious. But the governance decision, pause, classify, prove containment, then proceed, held.

What’s worth sitting with isn’t the science. It’s the decision structure. A group of people with genuine commercial and reputational incentives to move fast chose to build a classification framework before anyone made them. The pause was the control, not the obstacle to it.

Most boards I work with have no equivalent mechanism. Everything moves until something stops it. The Asilomar model inverts that. Nothing moves until the containment case is made.

The question it raises for AI governance is uncomfortable. Who in your organisation has the standing to call a pause on a deployment when the containment case hasn’t been made? And has that ever actually happened?

Disclosure is also a release decision

In 2008 Dan Kaminsky discovered a fundamental flaw in the DNS protocol. An attacker could poison a resolver’s cache and silently redirect internet traffic, corporate email, financial transactions, authentication systems, without the target knowing. The flaw wasn’t in one vendor’s implementation. It was in the protocol itself, which meant it affected nearly every operating system, ISP, and networking device on the internet simultaneously.

The standard response to a vulnerability discovery is disclosure. Find it, report it, publish it. Kaminsky didn’t do that.

He treated the release of information as a deployment decision.

He initiated a confidential multi-vendor coordination, Microsoft, Cisco, the Internet Systems Consortium, and kept the technical details under embargo while a synchronised patch was designed, tested, and prepared for simultaneous release across the ecosystem. The patch went out on 8 July 2008. The technical details followed after defenders had a head start.

The governance point isn’t about secrecy. It’s about sequencing. Kaminsky recognised that publishing a capability, even a defensive one, even accurate information about a flaw, was itself an act with consequences that needed governing. The question wasn’t whether to disclose. It was in what order, to whom, and when.

Most organisations don’t have a release governance process for information, only for software. They treat disclosure as a binary, tell people or don’t, rather than as a sequenced decision with timing, audience, and consequence built into it.

That distinction matters more as AI systems become capable of generating and distributing information at scale. The capability to produce convincing content at volume is already in most enterprise SaaS estates. Whether there’s a governance process for how and when it gets used is a different question.

When software reaches the physical world

In 2007 researchers at the Idaho National Laboratory demonstrated something that hadn’t been proven before. A remote attacker sending malicious commands to a digital control relay could physically destroy a 2.25-megawatt diesel generator. Not disrupt it. Not trigger a shutdown. Destroy it, in under three minutes.

The mechanism was straightforward once you understood it. Force the generator’s circuit breakers to open, wait for the rotating components to fall out of sync with the live grid, then close the breakers out of phase. The resulting electromagnetic stress tore the machinery apart. The attack required no physical access. No specialist equipment. Just commands sent over a network to a device that had no authentication requirement.

The Department of Homeland Security classified the findings immediately. Not because the vulnerability was exotic, it affected legacy industrial protocols used across global critical infrastructure. Because it was so replicable.

They sat on the technical details for seven years. During that time they worked quietly with power companies to install physical protective relays and hardware-enforced controls that would prevent the attack sequence regardless of what commands arrived over the network. The containment came before the disclosure, not after.

The governance decision here is the one worth noting. When the potential consequence is irreversible, physical destruction, cascading infrastructure failure, harm that can’t be walked back, the question changes. It stops being “can we release this” and becomes “what has to be in place before we do, and who decides when that threshold is met.”

That threshold question is the one missing from most AI governance frameworks. Organisations are asking whether a deployment is approved. Fewer are asking what the irreversible failure mode looks like, and whether the containment case has been made before that failure becomes possible.

AI is the current version, not the first version

In 2019 OpenAI developed GPT-2 and didn’t release it. Not immediately. Not fully. They were concerned that a model capable of generating convincing text at scale would be weaponised for disinformation and phishing before any defensive response was possible. So they released smaller, less capable versions first, monitored the ecosystem for signs of misuse, and staged the full release over several months.

It was Asilomar applied to model weights. Pause, classify the risk, prove the containment case, then proceed in stages.

The decision was controversial. Critics argued the risk was overstated. Others pointed out that comparable models would emerge from other labs regardless. Both were probably right. But the governance instinct, that releasing a capability is a decision with consequences that need sequencing, not just approving, was correct.

That instinct hasn’t scaled into most organisations deploying AI in 2026.

The frameworks exist. Anthropic publishes AI Safety Levels that trigger mandatory containment protocols as models reach higher capability thresholds. The EU AI Act creates a risk-tiered classification system with deployment conditions attached to each tier. The NIST AI Risk Management Framework provides a structured approach to pre-deployment assessment. The architecture of Asilomar is visible in all of them — classify by risk, match containment to classification, prove the case before proceeding.

What’s missing in most mid-market organisations isn’t the framework. It’s the decision structure that makes the framework operational. Someone with standing to pause a deployment. A classification process that runs before dependency is built. An honest answer to the irreversible failure mode question before the system goes live.

GPT-2 was a $50 million research lab making a considered call about a capability they’d built themselves. The governance challenge for most organisations is different, they’re deploying capabilities built by someone else, embedded in software they bought for another purpose, at a speed that makes pre-deployment classification feel like friction rather than governance.

More recent reporting around Anthropic’s Mythos and Project Glasswing shows why this question is becoming current rather than theoretical. The reported concern is not simply that an AI system can help defenders find vulnerabilities. It is that the same class of capability may lower the time, cost, and skill needed to identify, chain, and act on weaknesses before normal remediation processes can absorb the shock.

That is the containment problem in its modern form. The issue is not whether the capability is useful. It clearly can be. The issue is who gets access, under what conditions, with what safeguards, and what happens if the same capability becomes more widely available.

The governance question

The pressure to deploy is real and it isn’t going away. Boards want AI capability visible in the business. Vendors bundle it into renewals. Competitors are moving. The commercial case for pausing to classify feels thin when the use case looks contained and the upside looks significant.

That pressure is exactly the condition under which every capability governance failure in this piece occurred. Stuxnet’s designers had operational objectives. The molecular biologists had commercial and reputational incentives to move fast. Kaminsky’s peers expected immediate disclosure. The power industry had spent decades building infrastructure on the assumption that physical separation was protection enough.

In each case the governance decision that mattered wasn’t whether to proceed. It was whether the people making the deployment decision had honestly answered what happens when the capability moves outside the boundary they drew for it.

Most haven’t. Not because they’re careless. Because the approval process isn’t designed to ask that question. It’s designed to assess the intended use case against acceptable tolerances and document the outcome. That’s a different exercise.

The question that’s actually missing from most AI governance frameworks is this. What happens when this capability is copied, connected, automated, redirected, or put to use by someone who doesn’t share the assumptions we made when we approved it?

Not as a theoretical exercise. As a go/no-go condition.

If your current process can answer that question with evidence — tested boundaries, mapped dependencies, a named owner for the failure mode, a mechanism to pause deployment if the containment case isn’t made — then your governance is doing something real.

If the answer is an acceptable use policy and a terms of service review, you’re documenting the intended use case and calling it governance. That’s the pattern every case in this piece has in common. And in every case, the capability didn’t read the policy.

Before you approve the next deployment

There are a small number of questions that need honest answers before any significant AI deployment proceeds. Not documented answers. Honest ones.

What can this system do beyond the use case you’ve approved? Not the marketing description. Not the vendor’s intended application. The actual capability, if the boundary you drew around it didn’t hold.

Who can copy it, connect it, extend it, or redirect it, and what stops them? If the answer relies on policy or terms of service rather than a technical or contractual constraint, the boundary is assumed, not enforced.

What does the irreversible failure mode look like? The Aurora decision, seven years of quiet remediation before disclosure, was made because the answer to that question was physical destruction of critical infrastructure at scale. Most AI deployments don’t carry that consequence. But the question still needs an answer before deployment, not after the failure.

Would you know if it was being used in a way you didn’t intend? Anomaly detection tuned for capability misuse is a different instrument from standard uptime monitoring. If your observability tells you the system is running but not how it’s being used, the visibility gap is at the boundary that matters most.

Can you pause it, restrict it, or remove access if the risk changes? Not theoretically. In practice, without breaking something the business now depends on. If the answer is no, the deployment decision has already been made permanent by dependency.

And finally, who made the release decision, and what evidence did they accept before approving it? Not who signed the form. Who owns the answers to the questions above, and what did they actually verify.

The organisations that handle this well aren’t the ones with the most sophisticated frameworks. They’re the ones where someone had standing to ask these questions before the dependency was built, and the process had teeth enough to delay deployment until the answers were credible.

Which of those questions would your current approval process actually answer?

Two things happened after this piece was finalised.

On 10 June, Anthropic wrote to the US Senate Banking Committee alleging that operators affiliated with Alibaba’s Qwen lab had run 28.8 million interactions with Claude through approximately 25,000 fraudulent accounts between April and early June 2026. No jailbreak. No stolen credentials. They used the product as designed, at a scale that was never intended, and used the outputs to train a competing model. The boundary was the terms of service. It didn’t hold.

Two days later, the US Department of Commerce imposed export controls on Fable 5 and Mythos 5. Any foreign national, including Anthropic’s own employees, required an approved licence to access either model. Anthropic had no mechanism to screen users by nationality in real time. It disabled both models globally. Not a targeted capability restriction. A full shutdown, because the infrastructure couldn’t enforce a narrower one.

Both Mythos 5 and Fable 5 have since been partially restored for a named list of approved entities.

Both events follow the same shape this piece describes. Nobody had asked what happens when the capability moves outside the boundary drawn for it. In one case the answer was industrial-scale extraction by a commercial competitor. In the other, a blunt external control applied where a more precise one wasn’t available.

Acceptable Risk (Documented)

References

1. Asilomar Conference on Recombinant DNA — Wikipedia

https://en.wikipedia.org/wiki/Asilomar_Conference_on_Recombinant_DNA

2. Asilomar Conference (1975) — Embryo Project Encyclopedia

https://embryo.asu.edu/pages/asilomar-conference-1975

3. Aurora vulnerability: origin, explanation and solutions — INCIBE-CERT

https://www.incibe.es/en/incibe-cert/blog/aurora-vulnerability-origin-explanation-and-solutions

4. DNS Cache Poisoning Vulnerability (Kaminsky, 2008) — IANA presentation

https://www.iana.org/about/presentations/davies-cairo-vulnerability-081103.pdf

5. Melting the DNS Iceberg: Taking over your infrastructure Kaminsky style — SEC Consult

https://sec-consult.com/blog/detail/melting-the-dns-iceberg-taking-over-your-infrastructure-kaminsky-style