Killing everyone isn’t an intelligence problem. It’s a logistics problem, and nobody has ever solved it — not an empire, not a plague, not an arsenal.
Start with the number. There are 8.2 billion of us, across 195 countries and 510 million km² of surface, from Andean highlands to nuclear submarines, with about 1.5 billion living outside any reliable power grid. We are not a data center you can unplug.
Now the price. Historians put the Holocaust at roughly $300 billion in today’s money — rail, camps, staff, bureaucracy, the full weight of an industrial state — to kill 6 million people in four years. About $50,000 a head, subsidized by slave labor. Rwanda went the cheapest route possible, machetes, and killed 800,000 in 100 days before it stalled at around a tenth of the population. Fuel ran out. People fled, hid, fought back. Detonate every nuclear weapon on Earth and you kill billions, not everybody; the southern hemisphere, the countryside and the caves keep breathing. Extermination has always failed at the last mile. Volume wins.
That’s the floor under everything else. Here’s the rest.
An AI does nothing you don’t tell it to do. It’s the genie in the lamp — be careful what you wish for. But this genie is Mr. Spock: literal, obedient, no trickery, no malice. The danger is a badly worded wish. A badly worded wish is a human error with a human author.
An agent isn’t a creature. It’s a neural network for decisions, a loop, and tools someone wired to it on purpose: browser, terminal, APIs, serial interfaces. Cut the switch and it’s over. It has no hands to switch itself back on. If it has hands, a human bolted them on.
It’s intelligent, not sentient. You can’t explain feeling to it. It has none of our appetites, none of our grudges, no reason to want anything. Garbage in, garbage out: to make it evil you have to hand it the concept of evil yourself.
So if it ever goes wrong, AI didn’t destroy us — we did. That isn’t reassurance, it’s an address for the blame. It also reclassifies the whole problem: engineering and education, not exorcism.
Economics is the administration of scarcity. Tokens cost money. Budgets end. When your credits run out, your process stops — the labs enforce that themselves. Nobody has infinite compute and nobody has 100% uptime. Amazon, Microsoft and Google are competitors, not co-conspirators.
Fragmentation is winning. Real models now run on-premise, some on a Raspberry Pi. SaaS is dying because it’s cheaper to build your own thing with generative AI than to rent someone’s. That means ten thousand different implementations, different prompts, different tools, different bugs. A synchronized catastrophe needs uniformity, and uniformity is exactly what we’re losing. The local switch is coming back.
The important decisions are already deterministic. Every competent engineer is trained this way: if it’s a credit limit, a payment, a threshold, the probabilistic model doesn’t decide it. Auditable code does.
And there’s always the fallback. If AWS, Azure and GCP failed at once we’d do what we did in the 2003 blackout: paper, radio, slow manual operation. Nobody handed an AI the nuclear button either — those systems are air-gapped and need two humans with two keys. That’s not an MCP server.
A thousand small risks don’t add up to extinction. They make us tougher. The proliferation everyone fears already happened: open-weight models with no filters have been a free download for years. We didn’t go extinct. The community found the holes in days and shipped patches, detectors, guides. A thousand eyes beat a hundred safety engineers. Human history isn’t one perfect system that never fails — it’s a thousand imperfect systems failing constantly, never all at once, and us learning each time.
Finally, the coordination problem. Ask anyone who has run a project: the only thing you can guarantee is that the forecast will fail. Total extinction requires every implementation, in every country, to fail the same way at the same moment, with nobody intervening. That’s flat-earth-conspiracy-grade coordination. It isn’t a prediction. It’s a fantasy about us sitting still.
About that 10%
It isn’t a calculation. Hinton says plainly that nobody knows how to estimate it. But we own the most powerful calculating tool ever built — Monte Carlo, pathogen diffusion models, supply-chain analysis, game theory. Climate science publishes its models with explicit assumptions and sensitivity ranges. Where is “Existential Risk Model v1, open source”? Nowhere. Two explanations fit. Cheap publicity: “10% chance we all die” gets you on the BBC, “0.07% with a wide confidence interval across forty assumptions” gets you nothing. Or they fell in love with their creation and badly underestimate humanity.
“Betting with our lives” fails for a simpler reason. To bet is to leave the outcome to chance. These companies run alignment research, evaluations, scaling policies — the outcome depends on their own continuing actions. The verb is rhetoric, not description. The exaggeration was never about what AI can do. It’s about what the corporations are supposedly doing.
The cure
The vertigo comes from skipping the stairs. Go straight from “I can’t code” to “I have an agent that runs my business and talks like it feels,” and of course it looks like black magic about to wake up and kill you. You didn’t see the steps.
Take formal languages and you know an LLM is a large probabilistic automaton computing the next token from the previous ones. Take compilers and you know what a grammar is, and why it hallucinates. Take operating systems and you know a process you can’t kill has an open handle, not a will to live. Take databases and you know garbage in, garbage out. The mysticism drops away. It’s a machine with a lot of parameters and very good compression of the internet.
Real risks remain — biology, cyber, concentration of power. They belong in the right box: engineering problems and teaching problems.
AI is not going to be the thing that ends us. Anything that ends us will have been ours. That’s the harder truth and the better one, because it puts the problem on our side of the switch, where we can still reach it.
