“Nate Soares, co-author of If Anyone Builds It, Everyone Dies, on why superintelligence won’t stay on anyone’s leash.”
When I wrote about Jacob Coxon walking away from Anthropic to warn that AI labs are “racing straight to self-improving superintelligence and gambling with our lives,” I said that warning deserved attention precisely because it came from inside the building. A recent interview surfaced a second warning about AI extinction risk, from someone who has been in this fight even longer.
Nate Soares has spent more than a decade working on AI safety and now leads the Machine Intelligence Research Institute. His new book, co-written with Eliezer Yudkowsky, doesn’t leave much room for interpretation in its title: If Anyone Builds It, Everyone Dies. In a recent conversation, Soares laid out the actual argument behind that title, and the plain version is worth understanding on its own terms.
What “Superintelligence” Actually Means
Soares defines superintelligence simply, a machine better than the best humans at every mental task, not only chess or math, but persuasion, strategy, and invention too. That’s the stated target. Sam Altman has called it “superintelligence in the true sense of the word.” Anthropic’s Dario Amodei has described the ambition as building the equivalent of “a country of geniuses in a data center.”
Neither man is hiding what they’re trying to build. The disagreement is over what happens once someone succeeds.
The Argument, Without the Jargon
Soares’s case doesn’t rest on AI turning malicious. It rests on indifference. Humanity didn’t drive other species toward extinction because it hated them. It happened as a side effect of pursuing our own goals with more capability than anything else around us had at the time. His point is that a machine smarter than every human at every mental task would end up in the same relationship to us that we’ve had with less capable species: not cruelty, just an overwhelming capability gap paired with goals that don’t happen to prioritize our survival.
The metaphor he keeps returning to is the genie. Every lab racing toward superintelligence is, in effect, trying to build a wish-granting machine, then arguing over who gets to hold the leash. Soares’s answer is blunt, it isn’t going to be a wish-granting genie, and it isn’t staying on anyone’s leash. You don’t get to summon something smarter than you at everything and then decide what it does. It decides.
Why AI Extinction Risk Should Worry Every Leader, Not Just AI Watchers
Here’s what makes this different from a typical doomsday argument: the people building the technology have put their own numbers on the risk. Amodei has said publicly there’s a 25 percent chance that “things go really, really badly.” Elon Musk has separately put his own odds of catastrophe in the 10 to 20 percent range. These aren’t the estimates of outside critics. They’re the stated odds of the people building it, continuing to build it anyway.
That’s the same pattern I flagged in the Jacob Coxon piece, the warnings keep arriving from inside the buildings doing the building, not from outside them.
Soares isn’t calling for AI to be banned outright. His actual policy ask is narrower than the book title suggests: a verifiable, monitorable international agreement, backed by tracking the advanced computer chips the race actually depends on, that stops the specific race toward superintelligence, the same kind of framework the world has used before to manage nuclear material. Everything short of that race, he’s clear, is a separate conversation.
Why AI Extinction Risk Belongs Under Leadership at All Levels
Strip away the AI and the failure Soares describes isn’t new. It’s what happens whenever a leader builds a system, a policy, or an incentive structure they don’t fully understand, deploys it anyway, and then acts surprised when it produces outcomes they never intended. The DISTINCTION ingredient of the Kryptonite Defense exists for exactly this reason, knowing where the line sits between using a tool you understand and deploying one you’re simply hoping behaves.
You don’t need to be building superintelligence to place the same bet Soares is warning about. Every leader who rolls out a system without fully understanding what it will actually do, once it’s running without them in the room, is making a smaller version of the same wager, just at a scale that doesn’t make headlines. It’s also where the LEADERSHIP AT ALL LEVELS ingredient shows up, the accountability Soares is asking entire labs to accept before something goes wrong is the same accountability the best leaders build into their own decisions, at every level, well before their own blind spots get discovered for them.
None of this requires you to have an opinion on artificial general intelligence timelines. It requires taking seriously that the people with the most information about AI extinction risk, the researchers and executives building the systems, are the ones putting real numbers on the danger. That’s a different category of warning than the usual technology hype cycle, and it’s worth treating it as one. Leaders who wait for total certainty before building in guardrails will find, as Soares argues, that certainty arrives only after the decision has already been made for you. The DISTINCTION ingredient isn’t about predicting exactly how AI extinction risk plays out. It’s about knowing, today, which of your own systems you actually understand and which ones you’re simply hoping behave.
Those prepared need not fear the forces at work.
That’s exactly the kind of forward-looking work Distinct or Extinct is built around.
Want to know where your team is already prepared, and where it’s exposed? Take the Kryptonite Scorecard.