To Code is Awesome

In a large legacy system, writing a change may take a few hours. Finding where to make it, and proving that it won’t break anything, can take days or weeks. That difference explains almost everything the debate around AI agents tends to overlook.

The debate now has a reference text: Agentic Software: How AI Agents Are Restructuring the Software Paradigm. Its central thesis is clear. In traditional software, decision-making logic lives in the code. In agentic software, it lives in the agent, which generates code when it needs it. Code stops being the product and becomes a byproduct.

There is a lot of truth to that. Anyone who uses coding assistants sees it every day: increasingly, the value lies in expressing intent, in saying what you want to accomplish, and less in typing every line yourself. The paper is also honest about an important limitation: it acknowledges that agents perform very well on isolated tasks but still struggle with large projects, long-term maintenance, and systems that never stop changing.

Where it goes too far is in its tone. The title and several of its conclusions suggest something close to the end of software engineering. We have heard that promise before. Assembly to C. C to frameworks. Frameworks to low-code. Low-code to generative AI. Every new layer of abstraction was supposed to eliminate programmers, yet every time we still needed engineers. What changes is the level at which they work. Code does not disappear; it sinks one layer deeper, just as assembly is still there beneath everything we write in high-level languages.

So far, that is a reasonable and somewhat comfortable answer. The problem is that it treats “the industry” as if it were a single place.

Two Worlds with the Same Name

The prediction that in five years we will write far less code by hand sounds plausible. But it is plausible primarily for greenfield projects and startups. In a large enterprise, with massive legacy codebases spread across multiple platforms and technologies, and often poorly documented, there is still a great deal of manual interpretation to be done, along with plenty of code that still has to be written by hand.

Agents shine when a project starts from scratch, the architecture is simple or well documented, there are few external dependencies, and the requirements are clear. Now compare that with the typical landscape of a large enterprise. Millions of lines of code. .NET, Java, SAP, mainframes, Python, and JavaScript coexisting, not always peacefully. Business rules accumulated over decades. Incomplete or outdated documentation, which is sometimes worse than having none at all. Integrations with hundreds of systems. And, almost inevitably, behaviors that nobody fully understands but that the business depends on.

In that environment, the bottleneck is not writing. It is understanding.

Knowledge Engineering

When the main problem is figuring out what a system actually does and what will happen if you touch it, the work starts to look less like programming and more like investigation. It becomes knowledge engineering.

This is where AI does help, and quite a lot. It can explore code, explain modules, generate documentation, create tests, and suggest refactorings. All of that reduces the time engineers spend getting lost in someone else’s repository.

But there are things it still does not do well. It cannot reliably infer implicit business rules, the ones that were never written down because “everybody knew.” It cannot reconstruct historical decisions that nobody documented. It cannot assess organizational impact. And it cannot effectively navigate dependencies across dozens of teams, where the relevant question is often not technical but rather, “Who is this going to hurt?”

That is why automation moves faster in startups and new products than in organizations carrying decades of technical debt. It is not necessarily that large companies are slower to adopt new tools. It is that their most expensive work was never the work those tools accelerate.

What Actually Changes

None of this means the paper is describing a fantasy. The trend is real. In a few years, we will probably write far less code manually, even in large enterprises. What the paper conflates is the end of writing code by hand with the end of software engineering. Those are very different things.

What remains when code generation becomes automated is precisely what has always been difficult: architecture, systems design, security, governance, integration, product prioritization, and technical leadership. And there is another shift worth noticing. The more autonomous agents become, the more important it is to define objectives, constraints, and success metrics correctly. An agent that quickly executes a poorly framed instruction does not save work; it multiplies it.

In a legacy system, this matters even more. An agent may be able to write the change in minutes. What nobody can give it yet is the list of strange behaviors the business depends on, because that list does not exist in any file. It exists scattered across the minds of people who have been there for years, in old tickets, and in incidents someone vaguely remembers.

Code was always the easy part to explain. The hard part was, and still is, knowing what not to touch.

Killing everyone isn’t an intelligence problem. It’s a logistics problem, and nobody has ever solved it — not an empire, not a plague, not an arsenal.

Start with the number. There are 8.2 billion of us, across 195 countries and 510 million km² of surface, from Andean highlands to nuclear submarines, with about 1.5 billion living outside any reliable power grid. We are not a data center you can unplug.

Now the price. Historians put the Holocaust at roughly $300 billion in today’s money — rail, camps, staff, bureaucracy, the full weight of an industrial state — to kill 6 million people in four years. About $50,000 a head, subsidized by slave labor. Rwanda went the cheapest route possible, machetes, and killed 800,000 in 100 days before it stalled at around a tenth of the population. Fuel ran out. People fled, hid, fought back. Detonate every nuclear weapon on Earth and you kill billions, not everybody; the southern hemisphere, the countryside and the caves keep breathing. Extermination has always failed at the last mile. Volume wins.

That’s the floor under everything else. Here’s the rest.

An AI does nothing you don’t tell it to do. It’s the genie in the lamp — be careful what you wish for. But this genie is Mr. Spock: literal, obedient, no trickery, no malice. The danger is a badly worded wish. A badly worded wish is a human error with a human author.

An agent isn’t a creature. It’s a neural network for decisions, a loop, and tools someone wired to it on purpose: browser, terminal, APIs, serial interfaces. Cut the switch and it’s over. It has no hands to switch itself back on. If it has hands, a human bolted them on.

It’s intelligent, not sentient. You can’t explain feeling to it. It has none of our appetites, none of our grudges, no reason to want anything. Garbage in, garbage out: to make it evil you have to hand it the concept of evil yourself.

So if it ever goes wrong, AI didn’t destroy us — we did. That isn’t reassurance, it’s an address for the blame. It also reclassifies the whole problem: engineering and education, not exorcism.

Economics is the administration of scarcity. Tokens cost money. Budgets end. When your credits run out, your process stops — the labs enforce that themselves. Nobody has infinite compute and nobody has 100% uptime. Amazon, Microsoft and Google are competitors, not co-conspirators.

Fragmentation is winning. Real models now run on-premise, some on a Raspberry Pi. SaaS is dying because it’s cheaper to build your own thing with generative AI than to rent someone’s. That means ten thousand different implementations, different prompts, different tools, different bugs. A synchronized catastrophe needs uniformity, and uniformity is exactly what we’re losing. The local switch is coming back.

The important decisions are already deterministic. Every competent engineer is trained this way: if it’s a credit limit, a payment, a threshold, the probabilistic model doesn’t decide it. Auditable code does.

And there’s always the fallback. If AWS, Azure and GCP failed at once we’d do what we did in the 2003 blackout: paper, radio, slow manual operation. Nobody handed an AI the nuclear button either — those systems are air-gapped and need two humans with two keys. That’s not an MCP server.

A thousand small risks don’t add up to extinction. They make us tougher. The proliferation everyone fears already happened: open-weight models with no filters have been a free download for years. We didn’t go extinct. The community found the holes in days and shipped patches, detectors, guides. A thousand eyes beat a hundred safety engineers. Human history isn’t one perfect system that never fails — it’s a thousand imperfect systems failing constantly, never all at once, and us learning each time.

Finally, the coordination problem. Ask anyone who has run a project: the only thing you can guarantee is that the forecast will fail. Total extinction requires every implementation, in every country, to fail the same way at the same moment, with nobody intervening. That’s flat-earth-conspiracy-grade coordination. It isn’t a prediction. It’s a fantasy about us sitting still.

About that 10%

It isn’t a calculation. Hinton says plainly that nobody knows how to estimate it. But we own the most powerful calculating tool ever built — Monte Carlo, pathogen diffusion models, supply-chain analysis, game theory. Climate science publishes its models with explicit assumptions and sensitivity ranges. Where is “Existential Risk Model v1, open source”? Nowhere. Two explanations fit. Cheap publicity: “10% chance we all die” gets you on the BBC, “0.07% with a wide confidence interval across forty assumptions” gets you nothing. Or they fell in love with their creation and badly underestimate humanity.

“Betting with our lives” fails for a simpler reason. To bet is to leave the outcome to chance. These companies run alignment research, evaluations, scaling policies — the outcome depends on their own continuing actions. The verb is rhetoric, not description. The exaggeration was never about what AI can do. It’s about what the corporations are supposedly doing.

The cure

The vertigo comes from skipping the stairs. Go straight from “I can’t code” to “I have an agent that runs my business and talks like it feels,” and of course it looks like black magic about to wake up and kill you. You didn’t see the steps.

Take formal languages and you know an LLM is a large probabilistic automaton computing the next token from the previous ones. Take compilers and you know what a grammar is, and why it hallucinates. Take operating systems and you know a process you can’t kill has an open handle, not a will to live. Take databases and you know garbage in, garbage out. The mysticism drops away. It’s a machine with a lot of parameters and very good compression of the internet.

Real risks remain — biology, cyber, concentration of power. They belong in the right box: engineering problems and teaching problems.

AI is not going to be the thing that ends us. Anything that ends us will have been ours. That’s the harder truth and the better one, because it puts the problem on our side of the switch, where we can still reach it.