We’ll Pick That Up in the Next Release
Mid-career. An operational system going live.
“We’ll pick that up in the next release.”
The controls weren’t forgotten. They were scheduled. That’s a different problem.
Forgotten means oversight. Scheduled means someone looked at the risk, decided the timeline mattered more, and moved on. The downside still felt abstract. The go-live date didn’t.
I’ve been watching organisations make the same trade-off ever since. The language changes. The technology changes. The calculation doesn’t.
So when Professor Michael Wooldridge of Oxford warned recently that organisations face intense commercial pressure to deploy AI before its failure modes are properly understood. A single high-profile failure could trigger a sudden collapse in public confidence, the way the Hindenburg disaster ended the airship era overnight. I recognised it immediately. [1]
That’s why I wrote this. Not because the warning is new. Because I’ve already heard the sentence that comes before it.
“We’ll pick that up in the next release.”
Why AI Risk Feels Abstract Until It Isn’t
Infrastructure failures are noisy. A server goes down, an alert fires, someone gets a call at 2am. There’s a shape to it. A blast radius you can draw.
AI failures are quieter. And that’s what makes them harder to manage.
The system doesn’t go offline. It keeps running. It keeps producing outputs. The problem is that those outputs influence decisions. The gap between what the system said and what was actually true may not surface for weeks.
A Canadian tribunal case makes the point simply. An airline’s chatbot gave a customer incorrect refund information. When it went to arbitration, the airline was held liable for what its own interface had told someone. The chatbot didn’t malfunction. It just got it wrong, confidently, in a way that looked authoritative enough to act on. [4]
That’s the failure mode that doesn’t show up in your incident log.
Not “service unavailable.” Something closer to: the system worked fine, the output was wrong, and nobody knew until it mattered.
Now scale that to compliance guidance. Investment decisions. Clinical triage. Operational recommendations in a regulated environment.
The risk isn’t that AI breaks. It’s that AI is persuasive when it shouldn’t be certain.
Deployment Is Easy. Withdrawal Is Hard.
The other thing nobody talks about enough is how fast AI becomes impossible to remove.
Early technology mistakes were usually reversible. You could patch a system, segment a network, rebuild a server. The failure had edges. AI doesn’t work like that. By the time you find it, it’s already in the workflow.
I was working with a company recently. During the engagement it became clear that AI was already running through parts of the business: emails, marketing copy, internal documents. Some of it sanctioned. A lot of it not. People had just started using tools because the tools were there and the tools were useful. Nobody had said yes. Nobody had said no either.
That’s how it starts.
And once it’s embedded in how people work day to day, the conversation about governance gets much harder. You’re not deciding whether to adopt something new. You’re trying to retrofit control onto something that’s already load-bearing. According to a survey of over 1,700 data security professionals commissioned by Microsoft, 29% of employees have already turned to unsanctioned AI agents for work tasks. [3]
“We’ll govern it later” is what gets said at the start. By the time later arrives, switching it off has a cost. So it stays. And the governance conversation gets pushed to the next release.
Confidence Without Accuracy
There’s a specific characteristic of AI failure that doesn’t get enough attention. The systems are confident. Not accurate. Confident.
Wooldridge and others working in trustworthy AI have been flagging this for a while. Large language models don’t reliably signal when they’re uncertain. They produce fluent, authoritative-sounding output whether the underlying evidence is strong or whether they’re essentially guessing. CyBOK identifies this as one of the core unsolved challenges in AI assurance. Robustness and explainability remain genuinely hard problems, not marketing gaps that the next model release will fix. [2]
Which means the failure mode isn’t a system that crashes. It’s a system that sounds right.
Someone reads the output. It looks authoritative. It gets used. The error may not surface until it’s already been acted on. In a client document, a compliance summary, a board report. By that point the question isn’t whether the AI got it wrong. It’s who signed off on it and whether they can show they checked.
That last part is where most organisations currently have nothing.
The Same Playbook, Different Technology
The board pressure to adopt AI is real. I’m not dismissing it. Genuine efficiency gains are available. Customer expectations are shifting. Organisations that apply these tools well to pricing, fraud detection, forecasting, and operational work will build advantage over time. That’s not hype, that’s just true.
But a lot of what’s driving urgency right now isn’t strategy. It’s the same thing that’s driven urgency in security for the last two decades.
Vendor noise.
If you’ve watched the security market for any length of time you’ll recognise the pattern. A new category emerges. Vendors flood in. The message is always the same: deploy now, everyone else already has, the threat is existential, the window is closing.
AI has the same energy right now. Every software vendor has an AI story. Every pitch deck has an AI slide. The implicit message is identical. Move fast or fall behind. Some of that pressure is justified. Most of it is manufactured.
The part that concerns me is what’s getting hidden in the rush. Speed is being purchased with governance debt. Poorly understood data flows, decision logic nobody can explain, regulatory exposure that hasn’t been properly assessed. The competitive advantage looks real on the slide. The liability doesn’t show up until later.
Usually around the time someone says “we’ll pick that up in the next release.”
Where the Hindenburg Moment Happens
If there’s a Hindenburg moment coming, I think it happens in healthcare.
I’ve worked in and around healthcare, helping secure parts of their supply chain. What struck me wasn’t the technology. It was the people operating it. The clinical staff I saw were keeping their heads above water. Making life-changing decisions every day, under pressure that most organisations would consider a crisis and healthcare just calls Tuesday.
That’s the environment vendors are now marketing AI triage and decision-support tools into.
I understand why. The capacity pressure is real, the demand is relentless, and if AI can genuinely help then it should. But there’s a difference between AI that supports an experienced clinician and AI that substitutes for one. That distinction is getting blurred fast, partly because the people buying the tools are under enough pressure that anything promising relief looks attractive.
Research presented at the European Society for Emergency Medicine congress in September 2025 put numbers on that gap. AI accuracy in triage scenarios came in at 50.4%. Nurses scored 65.5%. Doctors reached 70.6%. The researchers’ conclusion was that AI could support triage alongside clinical staff, but should not be used as a standalone tool. [7]
That gap matters more in healthcare than anywhere else, because the failure mode isn’t a wrong invoice or a misdirected email.
It’s avoidable harm.
The accountability question in clinical AI isn’t new. Research published in the Bulletin of the World Health Organization in 2020 was already calling for safety case frameworks applied specifically to AI in healthcare, arguing that accountability and transparency needed to be built in before deployment, not retrofitted after an adverse event. [11] Five years on, most clinical AI deployments still don’t meet that bar.
When that case becomes public, the reaction from regulators, boards, and the media won’t be measured. It’s when, not if. The Hindenburg didn’t just ground one airship. It ended an era. That’s what Wooldridge is warning about. I think he’s right.
What the Frameworks Never Capture
I’ve been in enough incidents to know that the frameworks never really capture what the early hours feel like. The hard part is rarely the technology. It’s the human reality around it.
Nobody experiences it as a clear incident at first. It starts as ambiguity. Someone says it’s probably nothing. Someone else says we’ve seen this before. A few people quietly start checking whether this might actually be the bad version.
Very quickly you’re running on two clocks. The technical clock, where engineers want evidence before saying anything definitive. And the organisational clock, where leaders want to know the impact, the exposure, and whether the board or regulators need to hear about it. Those two clocks don’t run at the same speed and nobody warns you about that in the playbook.
Ownership turns out to be less clear than everyone thought. Systems depend on things nobody documented properly. The logs people assumed would tell the story are incomplete or missing entirely.
Then the tone in the room shifts. Speculation gets careful. Legal and comms appear. The question stops being “what broke?” and becomes “what can we prove?”
The human dynamics surface underneath all of it. Engineers worry they’re being blamed. Executives are thinking about customers and regulators. Long-standing organisational tensions become visible under pressure. Fatigue sets in. Early theories get sticky. It’s dangerously easy to mistake “we haven’t found more yet” for “it’s contained.”
What an incident actually reveals is the organisation’s real operating model. Whether ownership is clear. Whether teams trust each other. Whether evidence exists when it matters. Whether the risks everyone quietly knew about have finally arrived all at once.
Now add AI into that picture. No clear inventory of what’s deployed. No named owner for each system. No documented decision trail showing what the AI influenced and what a human verified. The question “what can we prove?” becomes very hard, very fast.
That’s not a technology problem. That’s a governance problem that was always there.
Governed Organisations vs Ungoverned Ones
The difference between governed and ungoverned organisations becomes obvious fast in those early hours. Not in the planning documents. In the room.
Governed organisations can answer the basic questions quickly. What AI systems do we have running? Who owns each one? What did they influence and when? What logs exist? What was verified by a human and what wasn’t?
Ungoverned organisations produce explanations.
Boards don’t accept explanations. Regulators don’t accept explanations. They ask for evidence. And if the evidence trail doesn’t exist, the explanation, however reasonable it sounds, fills the gap where accountability should be.
I’ve seen organisations with excellent policy documentation completely unable to answer basic operational questions under pressure. The policy said the right things. The reality hadn’t kept up.
That gap is where AI makes everything harder. Because the question isn’t just “what happened to our systems?” It’s “what did our AI influence, in which decisions, based on what inputs, reviewed by whom?”
Most organisations currently cannot answer that. Not because they’re negligent. Because nobody asked them to build the evidence trail before the incident arrived.
The Question Worth Asking
Here’s a question worth asking in any organisation that says it’s using AI responsibly.
Not “do you have a policy?” Not “have you done training?” Something more specific than that.
“Show me a real example where an AI output changed something in the business. And show me the evidence trail.”
What you’re looking for: the input or prompt, the output, where it went next, who reviewed it, what data it touched, which model produced it, what logs exist.
Not a presentation. Not a framework document. A real instance, traceable end to end.
If they can produce that quickly, they understand their systems. If they can’t. Most currently can’t. The gap has a name.
AI activity. Not AI control.
Those are different things. And the difference matters most at the moment nobody wants to be having the conversation about it.
A Regulatory Approach That Could Work
If regulators wanted to change behaviour quickly without killing innovation, I don’t think the answer is a longer rulebook.
There’s an established body of research worth bringing into the mainstream conversation. Academic work on AI safety cases, notably from Habli and colleagues at the University of York and from researchers at Google DeepMind, argues for exactly this approach: before a high-risk AI system goes into service, you build a structured argument, supported by evidence, that the system is acceptably safe for its defined purpose. [9, 10] The concept comes from safety-critical engineering. It’s been applied to medical devices and aviation software for decades. The researchers are now developing it specifically for AI.
The reason it hasn’t yet reached most boardrooms is simple. The work is happening in academic and frontier AI circles. It hasn’t crossed into the governance conversations where it’s most needed. That gap is what this section is about.
When a system is modified, upgraded, or deployed into a new environment, you don’t assume the original case still holds. You revalidate. The boundaries may have shifted. The failure modes may have changed. Accountability may no longer be clear.
Nobody in safety-critical industries finds that bureaucratic. Because the alternative is assuming everything is fine without checking, and that assumption tends to surface at the worst possible moment.
Before a system influences regulated outcomes, demonstrate four things. What it’s allowed to do and what it must never do. What data it was trained and tested on and how it behaves at the edges. Who owns it, across business, technical, and risk. And what the monitoring, audit, and rollback procedures are if it fails.
Then keep checking. Because the system that was validated six months ago may be running in a different environment today, connected to different data, used in ways nobody originally anticipated.
Show your working. And keep showing it.
That’s not a brake on innovation. Pilots and experiments can still run freely. But once a system starts affecting customers, markets, safety, or regulated decisions, it needs to be demonstrably controlled. The original safety case doesn’t expire at go-live. It expires the moment the system changes. And AI systems change constantly.
The Cybersecurity Angle Most Organisations Miss
There’s a cybersecurity angle to this that most governance conversations miss entirely. AI isn’t a standalone system sitting in a controlled environment. It gets wired into everything. And that creates attack surface that wasn’t in anyone’s original threat model.
The first signal I’m seeing in organisations right now is shadow AI creating uncontrolled data flows. People are connecting tools, Copilot, ChatGPT, Claude, smaller SaaS agents, into day-to-day workflows without thinking about what data is moving through them. Nobody approved that flow. Nobody mapped it. Nobody assessed what happens if that tool is compromised, or what the vendor does with the data, or whether it’s even compliant with the contractual obligations the organisation already has.
The second signal is vibe coding. Developers, and increasingly non-developers, are using AI to generate code fast, without fully understanding what it does. The code works, or appears to. It gets shipped. What doesn’t get shipped is any meaningful security review of what was actually built, what dependencies were pulled in, or what the failure modes are.
That’s how you get vulnerabilities in production that nobody can explain because nobody fully read what they deployed. Unit 42 research demonstrated that AI-assisted ransomware attacks can now move from initial compromise to data exfiltration in 25 minutes, roughly 100 times faster than the pre-AI baseline. The attack surface created by ungoverned AI deployments is the new entry point. [8]
Both of these are governance failures before they’re security failures. The technical risk is real. But it starts with the same decision that started this article.
Move fast. Pick it up in the next release.
Which Release Are You Currently On?
I’m not arguing against AI adoption. The efficiency gains are real. The competitive pressure is real. Organisations that get this right will build genuine advantage.
But I keep coming back to that meeting. Mid-career. An operational system going live. Someone in the room, not a bad person, not a reckless one, looked at the controls and said:
“We’ll pick that up in the next release.”
That sentence is everywhere right now. It’s in every organisation deploying AI without an inventory. Every workflow where the shadow tools are running and nobody has mapped the data flows. Every board that has approved an AI strategy but couldn’t tell you who owns each deployment or what the rollback plan is.
The next release always comes. The question is what arrives with it.
Sometimes it’s the governance catch-up that was always planned. Sometimes it’s the incident that makes the catch-up irrelevant.
The difference between those two outcomes isn’t the sophistication of the model. It isn’t the size of the AI budget. It’s whether someone in the organisation asked the hard questions before go-live rather than scheduled them for later.
Wooldridge is warning about a Hindenburg moment. I think he’s right to. I’ve seen the organisational conditions that produce it. I’ve heard the sentence that precedes it.
The question worth asking right now, honestly, is which release are you currently on. And what got scheduled for the next one?
References
4. Moffatt v. Air Canada, 2024 BCCRT 149. British Columbia Civil Resolution Tribunal, 14 February 2024.
6. BioCatch (2024). 2024 AI, Fraud and Financial Crime Survey. Survey of 600 fraud professionals.