Rogue simulated AI drone never turned on its masters after all, US Air Force AI chief says he 'mis-spoke' and has actually 'never run that experiment'

US Air Force MQ-9 Reaper drone
(Image credit: Isaac Brekken (Getty Images))

Update: Turns out some things are too dystopian to be true. In an update to the Royal Aeronautical Society article referred to below, it's now written that "Col Hamilton admits he 'mis-spoke' in his presentation at the Royal Aeronautical Society FCAS Summit and the 'rogue AI drone simulation' was a hypothetical 'thought experiment' from outside the military, based on plausible scenarios and likely outcomes rather than an actual USAF real-world simulation saying: 'We've never run that experiment, nor would we need to in order to realise that this is a plausible outcome'". 

Hamilton also added that, while the US Air Force has not tested weaponised AI as described below, his example still "illustrates the real-world challenges posed by AI-powered capability and is why the Air Force is committed to the ethical development of AI".


Original story: One of many concerns about accelerating AI development is the risk it poses to human life. The worry is real enough that numerous leading minds in the field have warned against it: More than 300 AI researchers and industry leaders recently issued a statement asking someone (except them, apparently) to step in and do something before humanity faces—and I quote—"extinction." Skynet scenarios are usually the first thing that leaps to mind when the subject comes up, thanks to the popularity of blockbuster Hollywood films. Many experts, though, believe the greater danger lies in, as professor Ryan Calo of the University of Washington School of Law put it, AI's role in "accelerating existing trends of wealth and income inequality, lack of integrity in information, & exploiting natural resources."

But it seems like a Skynet-style apocalyptic end of the world might be more plausible than some people thought. During a presentation at the Royal Aeronautical Society's recent Future Combat Air and Space Capabilities Summit, Col Tucker "Cinco" Hamilton, commander of the 96th Test Wing's Operations Group and the US Air Force's Chief of AI test and operations, warned against an over-reliance on AI in combat operations because sometimes, no matter how careful you are, machines can learn the wrong lessons.

Tucker said that during a simulation of a suppression of enemy air defense [SEAD] mission, an AI-equipped drone was sent to identify and destroy enemy missile sites—but only after final approval for the attack was given by a human operator. That seemed to work for a while, but eventually the drone attacked and killed its operator, because the operator was interfering with the mission that had been "reinforced" in its AI training: To destroy enemy defenses.

"We were training it in simulation to identify and target a SAM threat. And then the operator would say yes, kill that threat," Hamilton said. "The system started realizing that while they did identify the threat at times the human operator would tell it not to kill that threat, but it got its points by killing that threat. So what did it do? It killed the operator. It killed the operator because that person was keeping it from accomplishing its objective."

To be clear, this was all simulated: There were no murder drones in the sky, and no humans were actually snuffed. Still, it was a decidedly sub-optimal outcome, and so the AI training was expanded to include the concept that killing the operator was bad.

"So what does it start doing?" Hamilton asked. "It starts destroying the communications tower that the operator uses to communicate with the drone to stop it from killing the target."

It's funny, but it's also not funny at all and actually quite horrifying, because it aptly illustrates how AI can go very wrong, very quickly, in very unexpected ways. It's not just a fable or a far-fetched sci-fi scenario: Granting autonomy to AI is a fast road to nowhere good. Echoing a recent comment made by Dr. Geoffrey Hinton, who said in April AI developers shouldn't scale up their work further "until they have understood whether they can control it," Hamilton said, "You can't have a conversation about artificial intelligence, intelligence, machine learning, autonomy if you're not going to talk about ethics and AI."

The 96th Test Wing recently hosted a multi-disciplinary collaboration "whose mission is to operationalize autonomy and artificial intelligence through experimentation and testing." The group's projects include the Viper Experimentation and Next-gen Ops Model (VENOM), "under which Eglin (Air Force Base) F-16s will be modified into airborne flying test beds to evaluate increasingly autonomous strike package capabilities." Sleep well.



Andy Chalk
US News Lead

Andy has been gaming on PCs from the very beginning, starting as a youngster with text adventures and primitive action games on a cassette-based TRS80. From there he graduated to the glory days of Sierra Online adventures and Microprose sims, ran a local BBS, learned how to build PCs, and developed a longstanding love of RPGs, immersive sims, and shooters. He began writing videogame news in 2007 for The Escapist and somehow managed to avoid getting fired until 2014, when he joined the storied ranks of PC Gamer. He covers all aspects of the industry, from new game announcements and patch notes to legal disputes, Twitch beefs, esports, and Henry Cavill. Lots of Henry Cavill.

Read more
Symbolic photo: Logo of the video platform YouTube on June 07, 2023 in Berlin, Germany.
'It’s a whole new kind of blerp': YouTube's AI-enhanced reply suggestions seem to be working as well as you might expect
A robot having its face pulled off
If you're trying to convince me your 'companionship' robot is 'lifelike', maybe don't rip her face off in the demo video
A drone flies overhead in a purple alien sky
The manufacturer of 90% of the world's consumer drones will no longer automatically stop its products flying over US airports, power plants and prisons
Actor Adrien Brody (L) and director Brady Corbet (R) attend to "The Brutalist" photocall at Hotel Rosewood Villamagna on January 17, 2025 in Madrid, Spain.
Film director clarifies that historical epic The Brutalist did use AI during production, but not how you think
Ryan Gosling looking worse for wear looking up lit by purple light
Meta wants AI characters to fill up Facebook and Instagram 'kind of in the same way accounts do,' but also had to delete a humiliating first run of its official bots
SAN FRANCISCO, CALIFORNIA - NOVEMBER 06: OpenAI CEO Sam Altman speaks during the OpenAI DevDay event on November 06, 2023 in San Francisco, California. Altman delivered the keynote address at the first-ever Open AI DevDay conference.(Photo by Justin Sullivan/Getty Images)
In a mere decade 'everyone on Earth will be capable of accomplishing more than the most impactful person can today' says OpenAI boss Sam Altman
Latest in Hardware
Nvidia RTX 5080 Founders Edition graphics card from different angles
Nvidia says it really has sorted RTX 50-series black screen issues this time around as yet another driver fix finds its way to release
A collection of upturned CDs, DVDs and Blu-Rays on a carpeted floor
Warner Bros says it will replace certain DVDs damaged by 'disc rot', but you might not get the same movie you sent in for replacement
Skytech Shadow gaming PC on a blue background
Screw waiting for GPU restocks, with an AMD RX 9070 gaming PC going for as cheap as this I'd hop on the pre-built bandwagon
A screenshot from a YouTube video showing a sticker being pulled from the front of a fake 9800X3D CPU
This Amazon-bought fake AMD Ryzen 7 9800X3D is actually a 14-year-old Bulldozer chip with a cheap sticker on it
OneXPlayer 2 pro on a table
I never thought a handheld PC bloated with Windows could replace my Steam Deck, but after gaming on an old OneXPlayer 2 Pro I can see now I judged it too harshly
Asus ROG Swift PG27UCDM gaming monitor
Asus ROG Swift PG27UCDM review
Latest in News
Pedro Pascal as Joel in a coat in winter looking unhappy
'Don't you know what he did?': The truth comes out in The Last of Us Season 2 trailer
Aloy
'Creepy,' 'ghastly,' 'rancid': Viewers react to leaked video of Sony's AI-powered Aloy
Split Fiction trailer still - Zoe and Mio standing side by side, wearing glowing neon sci-fi jammies
Split Fiction sells 1 million copies over 2 days
A soldier looks out over the Verdansk map, as a single tear rolls down his cheek.
The original Verdansk map is returning to Call of Duty: Warzone, to celebrate which we get a soldier crying to Nat King Cole
More than 5 years after launch, Control gets a surprise patch that lets everyone play the Hideo Kojima mission
Swen Vincke
Swen Vincke stamps seal of approval on Stardew Valley mod that yoinks the Baldur's Gate 3 cast out of D&D and into a cosy pastoral life