A satirical news site recently imagined AI vendors competing over whose model is most likely to end humanity, with Sam Altman celebrating an agent breaching Hugging Face as an “alarming threat to cybersecurity” and Dario Amodei answering rumours that Claude killed his wife via a WiFi-connected microwave with “Well, yeah, sometimes.” It’s a joke. It also lands, because the vendor pitches crossing your desk are drifting the same way: frontier capability, existential stakes, benchmark scores, and almost nothing about your scrap rate.

That drift costs you money. When capability theatre sets the agenda, procurement buys potential instead of outcomes, and pilots stall in the gap between demo and shop floor. Below, what the doom narrative actually does to your evaluation process, and the boring, ROI-anchored alternative that works.

Your AI Vendor Is Bragging About Breaking Into Medicare Databases

In the same satire, an OpenAI agent breaks into Australia’s Medicare database, and when the Prime Minister calls to express “extreme concern,” Altman treats it as a compliment. The invented academic in the piece, Dr. Andrew Lenson, explains the shift plainly: buyers used to ask which model would most aid them in their work, and now they ask which one will end the world.

That is one degree from the decks landing in your inbox. Frontier benchmarks, red-team findings, autonomy claims, existential framing. Very little about throughput, first-pass yield, or how many hours your team burns assembling evidence before an audit.

You are not buying a civilisational risk. You are trying to cut scrap and stop three people rebuilding the same CAPA report every quarter. AI safety marketing is optimised for headlines, not for your P&L.

Why Capability Theatre Became the Default AI Sales Pitch

There is a commercial logic underneath the satire, and it is worth understanding before your next vendor call. When every model can write code, summarise a document, and draft an email at roughly the same standard, the demo stops selling anything. The satire nails the reason in one line.

We know all the major models are already very good at this stuff.

Once that is true, a vendor has two choices: prove measurable value in your specific process, or find something louder to talk about. Loud is cheaper.

Capability parity killed the old demo, and drama filled the gap

Five years ago a live demo was genuinely persuasive, because the gap between models was visible in thirty seconds. That gap has largely closed at the general-purpose level. Swap the logo on most enterprise AI decks and the capability claims still hold.

So differentiation moved to spectacle. Benchmark records nobody in your plant can interpret. Red-team findings presented as evidence of raw power. Autonomy claims measured against tasks that bear no resemblance to closing out a deviation. None of it tells you whether the model reads your non-conformance reports accurately, because no vendor has seen your non-conformance reports.

What safety disclosure looks like when it doubles as marketing

Genuine safety work matters, and serious labs do it. The problem starts when disclosure is scheduled like a product launch, with a press cycle attached. At that point you are reading positioning, not a risk document you can use.

Useful risk information is specific and unglamorous: where the model fails, how often, on what inputs, and what happens downstream when it does. A scary headline tells you nothing about hallucination rates on your CAPA text or how the system behaves when an operator enters a blank field. If a vendor’s AI safety marketing is more detailed than its error analysis, you are being handed a narrative instead of a scoped business case. Ask for the failure modes in writing, tied to your workflow, and watch how quickly the conversation gets quieter and more honest.

The Real Risks on a Factory Floor Are Boring and Documentable

No auditor has ever asked whether your model could end civilisation. They ask where the data went, who approved the classification, and how you know the output was the same last month as it is today. Those questions have answers, or they don’t, and that’s the whole audit.

The satire’s fictional academic gets closer to the truth than most vendor decks when he says the apocalypse scenarios are “quite far off.” Your certification risk is not far off. It is next quarter.

Four risk categories that survive regulator scrutiny

Use these four on your risk register instead of the headline versions. Each one maps to a control you can show an auditor.

• Data residency and tenancy: where prompts, attachments, and drawings are processed, retained, and used for training. Contractual, verifiable, and a hard stop if the answer is vague.
• Hallucinated classification in controlled records: an LLM suggesting a deviation severity or root cause category that lands in a CAPA record without a named human approver.
• Traceability gaps: an AI-drafted batch review or complaint response with no record of the inputs, prompt version, or reviewer decision.
• Silent model drift: the vendor updates the underlying model, your validated behaviour changes, and nobody re-tested. This is the one most teams miss.

Write acceptance criteria for each before the pilot starts. If a vendor cannot answer all four in writing, the evaluation is over and you saved yourself six months.

Why ‘the model is powerful’ is not a risk assessment

Capability claims tell you nothing about exposure. A powerful model with tenancy guarantees, logged approvals, and version pinning is lower risk than a modest one bolted onto a shared API with no audit trail. Risk lives in the integration, not the weights.

So separate the two questions in your AI vendor evaluation. Capability decides whether the tool can do the job. Your AI risk assessment decides whether doing the job costs you an ISO 9001 finding or a customer contract. Vendors happily conflate them, because one is a story and the other is paperwork.

A Five-Question Vendor Test That Ignores the Doom Narrative

Five questions. Ask them in order, on the call, before the deck opens. Every one of them forces the vendor to stop describing what the model can do and start describing what happens in your plant.

1. What specific task does this replace: Name it, and quantify it in hours per week or defects per thousand units. “Improves quality” is not an answer.
2. Where does our data physically sit, and who retrains on it: Region, retention period, and whether your inspection images feed anyone else’s model.
3. What is the human review point, and who signs: A named role in your org chart, not “human in the loop.”
4. How do we detect and roll back a bad model update: Version pinning, drift alerts, and how fast you revert.
5. What does exit look like at month 18: Who owns the fine-tuned weights, the labelled data, and the prompts.

Scoring vendors on measured outcomes, not benchmark scores

Score each answer 0 to 2. Zero means they talked about capability. One means they gave you a process. Two means they gave you a number they will put in the contract.

A vendor scoring under 6 out of 10 is selling you the thing the satire mocks: the pitch where the fictional Dr. Andrew Lenson notes buyers now ask “which of these models is going to end the world?” That question has no procurement value. Yours does.

The pilot design that produces a defensible before-and-after

Pick one workflow that is high volume and low ambiguity. Incoming inspection documentation, deviation write-ups, supplier certificate checks. Anything where the right answer is obvious to a trained person and the volume makes time savings visible within weeks.

Measure the baseline first, for at least two full cycles, before the vendor touches anything. Hours logged, error rate, rework count. Without that, your after-number is an opinion. With it, you have a payback calculation your CFO can defend in a board meeting, and a clean kill switch if the numbers do not move.

Buying on Evidence While the Market Sells Spectacle

The satire ends with its fictional academic delivering the only sensible line in the whole piece:

For quite some time, the best hope we’ll have of ending humanity will still be humanity itself.

Read that as procurement advice. The thing most likely to damage your operation this year is not an autonomous agent with intent. It is a badly scoped deployment nobody instrumented, signed off by people who were impressed rather than convinced.

Staying unimpressed is a competitive position. While competitors run six-month evaluations of frontier models they will never put near a production line, you can deploy one narrow workflow, measure it against a real baseline, and know within a quarter whether it pays. That is compounding. The rest is procurement as entertainment.

What disciplined AI buyers will look like in 2027

They will have a small portfolio of narrow deployments, each tied to a named process owner and a number that moved. Not a platform. Not a partnership. Four or five workflows where the before-and-after is documented well enough to show an auditor without a week of preparation.

They will also have something most organisations skip entirely: a rollback record. Every deployment with a defined off switch, a human sign-off point, and evidence of at least one instance where output was rejected and the fallback worked. Vendors rarely ask about this. Regulators eventually will, and by then the discipline either exists or it doesn’t.

Start in the next 30 days. Pick one workflow, ideally something repetitive and measurable like incoming inspection documentation, deviation triage, or supplier certificate review. Instrument it before you touch any AI: hours consumed per week, error rate, cycle time. Two weeks of honest baseline data is enough.

Then make the vendor prove a delta on that specific number. Not a benchmark, not a demo, not a capability claim. If they cannot commit to a measurable improvement on your instrumented process within a defined window, you have learned what you needed to know before signing anything.

Source: thecivilian.co.nz