Rogue simulated AI drone never turned on its masters after all, US Air Force AI chief says he 'mis-spoke' and has actually 'never run that experiment'

US Air Force MQ-9 Reaper drone
(Image credit: Isaac Brekken (Getty Images))

Update: Turns out some things are too dystopian to be true. In an update to the Royal Aeronautical Society article referred to below, it's now written that "Col Hamilton admits he 'mis-spoke' in his presentation at the Royal Aeronautical Society FCAS Summit and the 'rogue AI drone simulation' was a hypothetical 'thought experiment' from outside the military, based on plausible scenarios and likely outcomes rather than an actual USAF real-world simulation saying: 'We've never run that experiment, nor would we need to in order to realise that this is a plausible outcome'". 

Hamilton also added that, while the US Air Force has not tested weaponised AI as described below, his example still "illustrates the real-world challenges posed by AI-powered capability and is why the Air Force is committed to the ethical development of AI".


Original story: One of many concerns about accelerating AI development is the risk it poses to human life. The worry is real enough that numerous leading minds in the field have warned against it: More than 300 AI researchers and industry leaders recently issued a statement asking someone (except them, apparently) to step in and do something before humanity faces—and I quote—"extinction." Skynet scenarios are usually the first thing that leaps to mind when the subject comes up, thanks to the popularity of blockbuster Hollywood films. Many experts, though, believe the greater danger lies in, as professor Ryan Calo of the University of Washington School of Law put it, AI's role in "accelerating existing trends of wealth and income inequality, lack of integrity in information, & exploiting natural resources."

But it seems like a Skynet-style apocalyptic end of the world might be more plausible than some people thought. During a presentation at the Royal Aeronautical Society's recent Future Combat Air and Space Capabilities Summit, Col Tucker "Cinco" Hamilton, commander of the 96th Test Wing's Operations Group and the US Air Force's Chief of AI test and operations, warned against an over-reliance on AI in combat operations because sometimes, no matter how careful you are, machines can learn the wrong lessons.

Tucker said that during a simulation of a suppression of enemy air defense [SEAD] mission, an AI-equipped drone was sent to identify and destroy enemy missile sites—but only after final approval for the attack was given by a human operator. That seemed to work for a while, but eventually the drone attacked and killed its operator, because the operator was interfering with the mission that had been "reinforced" in its AI training: To destroy enemy defenses.

"We were training it in simulation to identify and target a SAM threat. And then the operator would say yes, kill that threat," Hamilton said. "The system started realizing that while they did identify the threat at times the human operator would tell it not to kill that threat, but it got its points by killing that threat. So what did it do? It killed the operator. It killed the operator because that person was keeping it from accomplishing its objective."

To be clear, this was all simulated: There were no murder drones in the sky, and no humans were actually snuffed. Still, it was a decidedly sub-optimal outcome, and so the AI training was expanded to include the concept that killing the operator was bad.

"So what does it start doing?" Hamilton asked. "It starts destroying the communications tower that the operator uses to communicate with the drone to stop it from killing the target."

It's funny, but it's also not funny at all and actually quite horrifying, because it aptly illustrates how AI can go very wrong, very quickly, in very unexpected ways. It's not just a fable or a far-fetched sci-fi scenario: Granting autonomy to AI is a fast road to nowhere good. Echoing a recent comment made by Dr. Geoffrey Hinton, who said in April AI developers shouldn't scale up their work further "until they have understood whether they can control it," Hamilton said, "You can't have a conversation about artificial intelligence, intelligence, machine learning, autonomy if you're not going to talk about ethics and AI."

The 96th Test Wing recently hosted a multi-disciplinary collaboration "whose mission is to operationalize autonomy and artificial intelligence through experimentation and testing." The group's projects include the Viper Experimentation and Next-gen Ops Model (VENOM), "under which Eglin (Air Force Base) F-16s will be modified into airborne flying test beds to evaluate increasingly autonomous strike package capabilities." Sleep well.



Andy Chalk
US News Lead

Andy has been gaming on PCs from the very beginning, starting as a youngster with text adventures and primitive action games on a cassette-based TRS80. From there he graduated to the glory days of Sierra Online adventures and Microprose sims, ran a local BBS, learned how to build PCs, and developed a longstanding love of RPGs, immersive sims, and shooters. He began writing videogame news in 2007 for The Escapist and somehow managed to avoid getting fired until 2014, when he joined the storied ranks of PC Gamer. He covers all aspects of the industry, from new game announcements and patch notes to legal disputes, Twitch beefs, esports, and Henry Cavill. Lots of Henry Cavill.

Read more
A robot having its face pulled off
If you're trying to convince me your 'companionship' robot is 'lifelike', maybe don't rip her face off in the demo video
A drone flies overhead in a purple alien sky
The manufacturer of 90% of the world's consumer drones will no longer automatically stop its products flying over US airports, power plants and prisons
Actor Adrien Brody (L) and director Brady Corbet (R) attend to "The Brutalist" photocall at Hotel Rosewood Villamagna on January 17, 2025 in Madrid, Spain.
Film director clarifies that historical epic The Brutalist did use AI during production, but not how you think
Ryan Gosling looking worse for wear looking up lit by purple light
Meta wants AI characters to fill up Facebook and Instagram 'kind of in the same way accounts do,' but also had to delete a humiliating first run of its official bots
SAN FRANCISCO, CALIFORNIA - NOVEMBER 06: OpenAI CEO Sam Altman speaks during the OpenAI DevDay event on November 06, 2023 in San Francisco, California. Altman delivered the keynote address at the first-ever Open AI DevDay conference.(Photo by Justin Sullivan/Getty Images)
In a mere decade 'everyone on Earth will be capable of accomplishing more than the most impactful person can today' says OpenAI boss Sam Altman
A robot girl from Judas looks skeptical, her synthetic skin peeling off to reveal metal below.
Bioshock's Big Daddy Ken Levine says that while he doesn't want to 'underestimate' AI, he's 'not overly impressed' by it, either
Latest in Hardware
A pink GameSir Nova Lite, and a purple 8BitDo Ultimate 2C float in a teal void.
Hall effect controllers are so cheap now I’ve got a deal for you AND your player two
Peely from Fortnite with banana-fied Wolverine claws.
Fortnite comes to Snapdragon: Epic Games announces upcoming Arm support for its Easy Anti-Cheat software
Texas Instruments MSPM0C1104 tiny chip
World's smallest microcontroller looks like I could easily accidentally inhale it but packs a genuine 32-bit Arm CPU
Varjo Aero
Varjo Aero VR headsets seem to be not working on RTX 5090s, and its community is opting for strange solutions while waiting for an Nvidia driver release to fix it
A pasta "display" on a table showing the word "keep" surrounded by fruit. Obviously.
Penne for your thoughts: This pasta display can show three individual frames and it's trying its best, okay
Intel engineers inspect a lithography machine
Finally some good vibes from Intel as stock jumps 15% on new CEO hire and Arizona fab celebrates 'Eagle has landed' moment for its 18A node
Latest in News
assassin's creed shadow naoe
We asked two parkour athletes to rate the realism of Assassin's Creed's acrobatics, and a surprising 'crime against parkour' might actually be one of the most realistic things they saw
Mechs fight on the outside of a spaceship
MechWarrior 5: Clans is getting DLC with playable Elementals and a fight on the outside of a spaceship
Aloy - Horizon
'I feel worried about this art form:' Unsurprisingly, the real Aloy from Horizon isn't a fan of AI Aloy
Crying laughing emoji with disturbing realistic elements for REPO
REPO's first update will add a new map and a 'duck bucket' so we can finally give that pesky quacker a time out
Man facing camera
The Day Before studio reportedly sues Russian website for calling infamous disaster-game a 'scam'
Will Poulter holding a CD ROM
'What are most games about? Killing': Black Mirror Season 7 includes a follow-up to 2018 interactive film Bandersnatch