Oops, ChatGPT can be hacked to offer illicit AI-generated content including malware

AI face.
(Image credit: Getty Images - Devimb)

Remember a while ago when OpenAI Ceo, Sam Altman, said that the misuse of artificial intelligence could be "lights out for all?" Well, that wasn't such an unreasonable statement considering now that hackers are selling tools to get past ChatGPT's restrictions to make it generate malicious content.

Checkpoint reports (via Ars Technica) that cybercriminals have found a fairly easy way to bypass ChatGPT content moderation barriers and make a quick buck doing so. For less than $6, you can have ChatGPT generate malicious code or a ton of persuasive copy for phishing emails. 

These hackers did it by using OpenAI's API to create special bots in the popular messaging app, Telegram, that can access a restriction-free version of ChatGPT through the app. Cybercriminals are charging customers as low as $5.50 for every 100 queries and giving potential customers examples of the harmful things that can be done with it. 

Other hackers found a way to get past ChatGPT's protections by creating a special script (again using OpenAI's API) that was made public on GitHub. This dark version of ChatGPT can generate a template for a phishing email to impersonate a business and your bank, even with instructions on where to best place the phishing link in the email.

Even scarier is that you can use the Chatbot to create a malware code or improve an existing one by simply asking it to. Checkpoint had written before about how easy for those without any coding experience to generate some fairly nasty malware, especially on early versions of ChatGPT, whose restrictions only became tighter toward creating malicious content.

Phising Email template generated by Telegram ChatGPT bot.

(Image credit: Checkpoint)

OpenAI's ChatGPT technology will be featured in the upcoming Microsoft Bing search engine update, which will feature an AI chat to provide more robust and easy-to-digest answers to more open-ended questions. This, too, comes with its own set of issues regarding the use of copyrighted materials.

We've written before how easy it's been to abuse AI tools like using voice cloning to make celeb soundalikes say awful things. So, it was only a matter of time before some bad actors found a way to make it easier to do bad things. You know, aside from making AI do their homework

Best SSD for gamingBest PCIe 4.0 SSD for gamingThe best NVMe SSDBest external hard drivesBest external SSDs


Best SSD for gaming: The best solid state drives around
Best PCIe 4.0 SSD for gaming: Speedy drives
The best NVMe SSD: Slivers of SSD goodness
Best external hard drives: Expand your horizons
Best external SSDs: Fast, solid, and portable

Jorge Jimenez
Hardware writer, Human Pop-Tart

Jorge is a hardware writer from the enchanted lands of New Jersey. When he's not filling the office with the smell of Pop-Tarts, he's reviewing all sorts of gaming hardware, from laptops with the latest mobile GPUs to gaming chairs with built-in back massagers. He's been covering games and tech for over ten years and has written for Dualshockers, WCCFtech, Tom's Guide, and a bunch of other places on the world wide web. 

Read more
SUQIAN, CHINA - JANUARY 27, 2025 - An illustration photo shows the logo of DeepSeek and ChatGPT in Suqian, Jiangsu province, China, January 27, 2025. (Photo credit should read CFOTO/Future Publishing via Getty Images)
The brass balls on these guys: OpenAI complains that DeepSeek has been using its data, you know, the copyrighted data it's been scraping from everywhere
One YouTuber has been poisoning AI tools that access her videos with .ass subtitle files and you can too
Closeup of the new Copilot key coming to Windows 11 PC keyboards
Microsoft co-authored paper suggests the regular use of gen-AI can leave users with a 'diminished skill for independent problem-solving' and at least one AI model seems to agree
The OpenAI logo is being displayed on a smartphone with an AI brain visible in the background, in this photo illustration taken in Brussels, Belgium, on January 2, 2024. (Photo illustration by Jonathan Raa/NurPhoto via Getty Images)
OpenAI is working on a new AI model Sam Altman says is ‘good at creative writing’ but to me it reads like a 15-year-old's journal
OpenAI logo displayed on a phone screen and ChatGPT website displayed on a laptop screen are seen in this illustration photo taken in Krakow, Poland on December 5, 2022.
New research says ChatGPT likely consumes '10 times less' energy than we initially thought, making it about the same as Google search
OpenAI representatives using a rotary phone to call ChatGPT via the 1-800-ChatGPT phone number
You can now WhatsApp message ChatGPT or call it on the phone, even from an old rotary blower. What a time it is to be alive
Latest in AI
Closeup of the new Copilot key coming to Windows 11 PC keyboards
Microsoft co-authored paper suggests the regular use of gen-AI can leave users with a 'diminished skill for independent problem-solving' and at least one AI model seems to agree
Still image of Bastion holding a bird, taken from Microsoft's Copilot for Gaming reveal trailer
Microsoft unveils Copilot for Gaming, an AI-powered 'ultimate gaming sidekick' that will let you talk to your console so you don't have to talk to your friends
BURBANK, CALIFORNIA - AUGUST 15: Protestors attend the SAG-AFTRA Video Game Strike Picket on August 15, 2024 in Burbank, California. (Photo by Lila Seeley/Getty Images)
8 months into their strike, videogame voice actors say the industry's latest proposal is 'filled with alarming loopholes that will leave our members vulnerable to AI abuse'
live action Jimbo the Jester from Balatro holding a playing card and addressing the camera
LocalThunk forbids AI-generated art on the Balatro subreddit: 'I think it does real harm to artists of all kinds'
Aloy
'Creepy,' 'ghastly,' 'rancid': Viewers react to leaked video of Sony's AI-powered Aloy
Seattle, USA - Jul 24, 2022: The South Lake Union Google Headquarter entrance at sunset.
Google is rolling out an even more AI-heavy search engine mode because 'power users want AI responses for even more of their searches'
Latest in News
Will Poulter holding a CD ROM
'What are most games about? Killing': Black Mirror Season 7 includes a follow-up to 2018 interactive film Bandersnatch
Casper Van Dien in Starship Troopers
Sony, which is making a Helldivers 2 movie, is also making a new Starship Troopers movie, but it's not based on the Starship Troopers movie we already have
Assassin's Creed meets PUBG
Ubisoft is reportedly talking to Tencent about creating a new business entity to manage Assassin's Creed and other big games
Resident Evil Village - Lady Dimitrescu
'It really truly changed my life in every possible way': Lady Dimitrescu actor says her Resident Evil Village role was just as transformative for her as it was for roughly half the internet in 2021
Storm trooper hero
Another live service shooter is getting shut down, this time before it even launched on Steam
Possibility Space concept art.
Possibility Space owners sue NetEase for $900 million over allegations it spread 'false and defamatory rumors' of fraud at the studio that ultimately forced it to close