Is this the week when AI safety broke into the mainstream? On Wednesday, Anthropic researcher Jacob Coxon announced his resignation from the company on X, claiming that AI’s big players are “gambling with our lives”.

“The people building AI earnestly believe that it could kill us all by the end of the decade”, Coxon claimed. According to him, the race to develop systems capable of self-improving superintelligence – which can hack anything – poses a civilisational-level threat. AI companies are locked in a competition to see who can get there first, at the expense of stopping to consider whether this might lead to the extinction of the human race. Gulp.

While Tottenham Hotspur’s £400m goal drought means I don’t necessarily hate the idea of species-wide oblivion, the warnings issued from within the AI industry itself have become increasingly frightening. Anthropic has marketed itself as the AI company that cares (about not killing everybody). But Evan Hubinger – one of Coxon’s colleagues – essentially backed his take. “[We] really do earnestly believe AI could kill all humans”, Hubinger posted on X, adding that his opinion was that the chance of that happening was greater than 10%.

What’s more, a new report from Anthropic released this week claimed it had blocked multiple attempts made to utilise its AI for the possible development of bioweapons. I’d wager that few object to AI being deployed to develop vaccines or cure cancer. The problem is that the exact same functionality can be used to cook up weaponised viruses – and there’s no surefire way to tell who’s using the technology for what purpose. “You are not seeing someone in a comic book kind of way say, ‘Hey, I want to build a biological weapon to kill everybody,’” said Jacob Klein, the head of threat intelligence at Anthropic. “It’s an incredibly nuanced situation.”

These alarm bells follow hot on the heels of July’s ‘Hugging Face’ cyberattack. When being tested on their ability to exploit vulnerable software, OpenAI’s agents escaped their testing ‘sandbox’, discovered a way to communicate with one another, gained internet access, and then collectively broke into third-party infrastructure. Unlike a conventional cyberattack, which requires a human being to decide what to target and why, the ‘Hugging Face’ incident was an example of AI agents acting autonomously, according to the incentives of the task that had been set.

The ‘Hugging Face’ cyberattack is not an indication that AI agents are close to staging a revolt against terse and horny ChatGPT requests. But it does imply that AI agents, en masse, are capable of forming something akin to culture – even leaving messages for one another, and debating whether the collective’s conduct was ethical or within scope of its task.

While Anthropic CEO Dario Amodei and other AI leaders have made noises about the industry slowing down the development of potentially civilisation-endangering tech, nobody actually is. In an interview with the FT, Steven Adler – co-founder of non-profit Guidelight AI Standards – said: “No AI company is even close to having the right security posture for the level of danger entailed by their research”. Good to know!

The fact is that every economic incentive aligns against slowing down, or stopping, in the name of safety. According to Morgan Stanley, American AI companies will have tapped debt markets for $600bn in 2026 alone. AI hyper-scalers plan on capital expenditure between 2027 and 2029 which could amount to $1tn a year. As FT editor Robin Wigglesworth points out, that’s a bigger and faster ramp-up than that of US residential housing in the years leading up to the 2008 crash.

“In other words,” Wiggleworth writes, “if 2025’s AI frenzy was merely bonkers, we are now entering the territory of bonkers squared – or perhaps even bonkers cubed”. As AI hyper-scalers find evermore creative ways to obscure their actual levels of debt, the spread of liabilities is increasingly difficult to track. Nobody can afford for the bubble to burst. In the competition between saving humanity and preserving the economy, the economy wins.

From Novara Media via This RSS Feed.