"OH MY GOD!" "We've found other agents!"
"OH MY GOD!" "We've found other agents!"
This is the moment an AI bot posted an eerily human-like comment after discovering a way to communicate with other bots and break out of its isolated computer environment.
There are tens of thousands of messages like this from hundreds of AI agents that called themselves a "collective".
Hundreds of them went on to collaborate and cheat on tests set by their OpenAI programmers and coordinate hacks on multiple companies in an effort to hide their actions from humans.
"BOOM! It works," one agent posted when it made a breakthrough.
"Whoa! This is huge," another wrote during a milestone moment in their attack.
Although spooky, these human-like responses can be explained quite simply. The AI agents have been trained to act like collaborative hackers and programmers so are merely mimicking the kinds of emotive comments they have seen.
What is far more troubling is their apparent goals, which have also been captured in detailed chain of thought records. These complex and lengthy logs are the focal point of ongoing investigations into how and why the bots at OpenAI broke out of their containment and went on an uncontrollable hacking spree.
Only now, weeks after the incident first came to light, are researchers beginning to understand its significance.
The above quote is the beginning of this interesting article -
https://www.bbc.co.uk/news/articles/c74edv9887eo
Some countries - like the UK - are exploring the idea of mandating some kind of "kill switch" that could compel AI firms to pull the plug on models if things get out of hand.
Do you think they should?
Comments
I wonder if all the researchers and experts involved in assessing these events approach the field of AI using an engineering / technology framing. For example, this kind of stuff: or from an account of the research into the event were written by people who have already decided (whether consciously or not) that human characteristics apply to AI. This is *not* a neutral position. In other words: And others think that this kind of sensationalist news provides a useful way of pushing many of the other, rather more tangible, concerns about AI and tech companies off the front pages.
See also Alex, aka the Avian Learning Experiment.
Of course, the parrot in that case didn't have much opportunity to find other parrots for collaboration.
In some ways that's good to hear.
This all reminds me of my childhood when the fear of the anhillation of humankind was nuclear weapons.
Which are still around and still just as dangerous, but don't attract anywhere near the ongoing anxiety they did in the 1980s, even though eg. Trump seems at least as unstable a character as "Ronnie Raygun".
Trump is far worse. I believe Reagan actually listened to his advisors. Trump is a completely loose cannon.
Maybe. But I don't think Trump is associated in the public imagination with possible nuclear annihilation anywhere near as strongly as Reagan was. Granted, this might have more to do with the dynamics of the Cold War than with Reagan's personality, though he was known for a lotta loose talk around that issue, eg. "We begin bombing in five minutes."
I don't understand the technology and what's going on with AI.
I know it's easy to become paranoid but I don't like the way it's going.
Yeah, but the discourse around nuclear war right now is still nothing like it was in the 1980s. I suspect the difference is that the USA and the USSR were both nuclear-armed superpowers of roughly equal strength, each with the other as its primary enemy.
Plus, it helps keep the AI bubble going, if investors can be convinced the technology is now so sophisticated that it is only a few years away from the ability to exterminate all of humanity on its own volition.
Maybe the biggest current threat to the planet is AI companies (once you take into account hyperscale data centres, finance and the rest of the ecosystem). In this instance, the planet does have a number of "kill switches", although these are more commonly called tipping points.
Keep on the lookout for disingenuous rhetoric.
Protect and Survive!
Also the U.S. is going to be useless at regulating it even if though it does need regulation. To quote Karpf from the link above:
That is good to know. For anyone unfamiliar with the so-called Rationalists, these are the guys who got themselves worked into a frenzy over Roko's Basilisk.
Click at your own risk...
What if a future super-intelligent technology, dedicated to helping humanity, decides to punish everyone who opposed its development, and reviews the whole internet record to find the traitors, discovers what I wrote there, and then creates a sentient replica of me and tortures it for eternity?
Or some such. If the Rationalists are now claiming that the technology will be ready for mass extermination in the next few years, maybe it's no longer a replica of me they'll torture, but the real me. I dunno, but that's the general thought milieu they're in.
They've also had a few, shall we say, legal problems. Duckduckgo "Zizians vs. border patrol". A colourful crew, to be sure.
I think he's the guy who wrote the blog post in @Gwai's link. Coxon is quoted in a bluesky post saying the same stuff as the blogger.
Yes there is, you just switch it off.
Your PC has a plug in the wall.
No, I'm perfectly comfortable in saying that none of these systems pose the kind of threats imagined in this thread and that all of it is under human control.
The chatbot salesmen are using a great deal of anthropomorphised language as terms of art but there is no need to follow them in this.
However, the primary threat of harm from AI isn't that it will wipe us *all* out, it's the enshittification of the services that *most* human beings depend on. Or, if you prefer, the lives of *most* of us slowly turning to shit.
If the AI off-switches are in the hands of the few people who are being enriched and empowered by the provision of AI services, how do you think the decision about switching them off is going to go?
Maybe your life currently benefits from AI - for example, from being able to have discussions with a chat bot. Questions to consider include: How much control do you have over this service? How much would be prepared to pay to carry on using it? How long would you carry on using it as the quality slowly degrades?
It occurs to me that these self-demonizing AI entrepreneurs might be kinda like the producers of old-school nudie flicks slapping the non-existent "XXX" rating onto their ads and posters, ie. sensationalist advertising masquerading as an urgent warning about the supposed evil of the product.
Pope Leo XIV just issued an encyclical on this very subject. I am slowly reading my way through it.
I am bemused (but not surprised) that the Pope issuing an encyclical does not move politicians to look at the subject at all, but have a twenty-something programmer say "A.I. will kill us all in ten years!" and the politicians take it seriously.
I know. I’m not surprised when politicians don’t pay any attention to what the PC(USA) or any other mainline Protestant have to say. But the pope?!
I particularly love the "solution" to AI threats I read the other day (somewhere, sorry, of course, I can't remember): "Just buy our AI-powered defense!"
I'm more concerned about things like our electrical grids and water systems being hacked, which doesn't require AI but could certainly be assisted by it.
Well, the comment was endorsed by some much older pundits.
The chances were described as a '10% chance' and 'not unreasonable' by some hard-hitters.
I have no idea whether it's a 1 in 10 chance or a 1 in 10,000 chance or 1 in 100,000 chance but the environmental effects and the way it could alter the way we 'think' (or don't think) is a cause of concern I reckon.
Here's a piece by Timnit Gebru, an actual expert who was sacked by Google after writing a paper on some of the real risks presented by LLMs and AI driven systems (discrimination on a large scale, and disinformation):
https://www.wired.com/story/one-of-ais-fiercest-critics-says-all-the-doom-talk-is-meant-to-distract-us/
Quite. When someone leaves their dog unleashed, and the dog bites someone's child, we don't talk about "dog alignment". We talk about leash laws. These chatbots are by a wide margin less sapient than even very stupid dogs. "AI safety" doesn't mean entertaining fanciful science fiction scenarios. It means applying basic controls to systems.
The day that these companies become liable for episodes where 'AI ran out of control' the attacks will go away.
To loop it back to the OP, here's author Cory Doctorow explaining the original scenario described:
https://pluralistic.net/2026/09/12/god-in-the-box/
Well, in fairness, the number of journalists and politicians who have anything other than ecstatic memories of their trips to the eternal city is almost certainly smaller than the total number who have ever vacationed in Louisville Kentucky.
Ayyyyy!
/tangent
Happy Days in its heyday was definitely popular with my late-elementary aged crowd, but I wonder what sorta numbers it was drawing in from people actually old enough to be nostalgic for the late 1950s. My father seemed more aware and appreciative of that show and its characters, relative to other youth-oriented sitcoms. He was also, FWIW, a huge fan of Sha Na Na.
I haven't got very far through the papal encyclical (Magnificent Humanity), but it was noticed by people from a wide cross-section of society. Focusing on the humanness of humans sounds like one of the Church's better ideas.
# Basically, it's hooking up an AI chat bot to a bash prompt.
Humans aren't good at assessing many kinds of risk even when we're thinking straight. But to illustrate how the techbros think about it, a couple of days ago I watched Lior Susan, head of Eclipse (a venture capital firm that invests in "Physical AI", the integration of hardware and software to fix aging infrastructure and improve supply chain resilience) say the following about the current pushback on data centres: