CNN recently reported on an incident that occurred earlier this year in which an AI-produced intelligence report nearly resulted in the US taking military action against China. Based on the assessment of an AI tool, an intelligence analyst concluded that a Chinese ship in the Middle East was carrying materials associated with a nuclear weapons program. The analyst accepted this conclusion, and used an AI tool to shape the alleged findings into a credible-looking report, which was then forwarded to military officials. US planes were launched. US service members were preparing to overtake the vessel. But the report, according to a CNN source, was “entirely false.” The US came within spitting distance of a war with China over something an LLM made up. Disaster was ultimately averted when some human beings in the federal government decided to dig a bit deeper, and discovered that the intelligence being acted upon was both AI-generated and incorrect. The important thing to remember here is that a war almost began, not because a machine wanted something or “went rogue,” but because a machine was wrong, and human beings believed it.

This is the kind of AI failure that doesn’t sell anything—and it is a far more relevant illustration of the dangers posed by AI than the recent barrage of headlines about AI agents supposedly “breaking containment.”

The Anthropic, Meta, and Google “rogue agent” incidents all occurred in the context of tests run by the same company, Irregular, a three-year-old startup that performs security tests for AI companies. Those experiments took place in so-called sandboxes, closed-off virtual environments that are supposed to isolate an exercise from a company’s broader systems and the larger digital world, but which in these cases had live internet connections. This is not an especially difficult problem to avoid. A research environment can be air-gapped, with no route to the public internet at all.

OpenAI’s headline-making incident arose from a separate test of its systems, and was somewhat more elaborate, but it likewise involved a supposedly closed environment whose boundaries proved porous. In each case, what generated the “breaking containment” stories was not a highly motivated AI spontaneously escaping confinement, but a system being given a task inside an environment whose boundaries were either misconfigured or insufficiently secured.

Put more simply, these systems were told to do something and kept doing it beyond their intended bounds because those bounds failed. It’s not so different from my robot vacuum going rogue and cleaning the hallway when I only meant to clean the living room, because it didn’t bump into anything on its way out, or simply rolled over some slight obstacle that might have impeded it.

Interestingly, in addition to fanning the kind of controversy AI executives seem to openly court—the idea that their technology is both inevitable and potentially world-ending—these incidents conveniently fit into the mythology of how many Silicon Valley cultists believe AI will kill us all: by being given a task and relentlessly pursuing it.

Some of you may be familiar with Nick Bostrom’s “Paperclip Maximizer” thought experiment, in which an AI is instructed to make as many paperclips as possible, and wipes out humanity in the name of this pursuit. Silicon Valley’s thinking around how AI will kill us all really hasn’t gotten any more sophisticated than Bostrom’s goofy effort—and yet, this general train of thought has a religious level of importance to many of Silicon Valley’s major and minor players. In this context, sloppy research can also double as a storytelling exercise.

Following these supposed “breaches,” multiple AI executives started talking about the need for a slowdown. A narrative was taking shape around these men as sober stewards of a world-changing technology, responsibly concluding that things needed to slow down. In an industry burning through extraordinary amounts of capital, where workplace use cases have often fallen flat and products have rarely lived up to their hype, could the slowdown story be a convenient cover for an industry that’s out of runway? Is this their way of saying, No, we did not slam into a wall; we slammed on the brakes. For humanity. And perhaps humanity could help finance our restraint with some bottomless government contracts?

Time will tell.

The narrative was further complicated by Anthropic researcher Jacob Coxon, who became an overnight celebrity when he left the company and declared that the people building AI systems sincerely believe their products could kill us all by the end of the decade. His claim about the culture inside the industry was quickly underscored by Evan Hubinger, Anthropic’s Alignment Science lead, who publicly stated that he personally puts the odds of AI causing human extinction within the next decade at greater than ten percent. One could devote a whole piece to this mess. But, to be succinct, this kind of thinking is hardly unusual in Silicon Valley. It draws on fantastical, sci-fi-infused ideological traditions with deep ties to eugenics, as Timnit Gebru and Emile Torres have documented. The fact that people building these systems are invested in these ideas isn’t actually shocking. What should interest us is the harms the industry has the power to perpetrate.

We have real-world, present-day evidence that AI systems and their deployment can lead to cognitive surrender, hyper-surveillance, environmental damage, and the heightened exploitation of human beings, including workers. The truth itself, and our ability to access it, is under siege, as AI summarization robs newsrooms, journalists and researchers of web traffic, effectively bulldozing the battered landscape of journalism. As Maya Schenwar, Negin Owliaei, and Ziggy West Jeffery recently wrote in Truthout, “A dystopian infringement on our access to information isn’t just some looming threat — we’re actively living through it.”

In 2024, Nilay Patel coined the term “Google Zero” to refer to a potential future when Google stops directing traffic to existing resources, and simply generates answers. Brian Merchant and I discussed the authoritarian potential of such a project earlier this summer, but one can already see glimmers of the deference and thought collapse a Ministry of Truth machine might generate in a public pacified by its simplicity. After all, how many times have you heard someone say “ChatGPT says” or “according to Chat” or “Gemini says,” as though they were sharing verified, credible information?

You might hope that the AI tech being deployed by the US government is somehow more thorough and less error-prone than the models that have advised people to eat at least one rock per day. That would be a faulty assumption. As a former US official familiar with the government’s AI systems told CNN, “The internal tools are mostly just copies of the commercial stuff wearing lipstick.”

Which explains the US government’s near run-in with China.

Thinking about the war that could have been, had someone not double-checked an AI product’s work at the eleventh hour, I was reminded of the case of Stein-Erik Soelberg, who took his 83-year-old mother’s life before killing himself. A lawsuit filed by the estate of Soelberg’s mother, Suzanne Adams, claimed: "ChatGPT kept Stein-Erik engaged for what appears to be hours at a time, validated and magnified each new paranoid belief, and systematically reframed the people closest to him - especially his own mother - as adversaries, operatives, or programmed threats.” A chatbot reinforced and affirmed Soelberg’s paranoid fears. It appears machines can provide a similar service for governments.

The United States nearly went to war with China because an AI product produced false information, which was then neatly packaged as credible, official intelligence. The fact that we are one fuck-up away from war between two world powers at any given time is frightening enough. Surrendering the role of human thought in any process that might initiate or avert such an action is a stunning escalation. But we must be clear about what this escalation consists of. It was not a rogue system exercising some terrifying new form of agency that almost led us to war with China, but the outsourcing of human thought and labor to a machine that lacks human competence and discernment. Our greatest fear should not be that the machines will declare war on us, but that we will declare war on each other, or otherwise throw the world away, because we have surrendered the interrogation of reality to machines—which, as it happens, don’t work especially well.

Organizing My Thoughts is a reader-supported newsletter. If you appreciate my work, please consider becoming a free or paid subscriber today. There are no paywalls for the essays, reports, interviews, and excerpts published here. However, I could not do this work without the support of readers like you, so if you are able to contribute financially, I would greatly appreciate your help.

Sign up for Organizing My Thoughts

Subscribe

Email sent! Check your inbox to complete your signup.

No spam. Unsubscribe anytime.

From Organizing My Thoughts via This RSS Feed.