Researchers build AI that can clone Pac-Man without a single line of code

YouTube YouTube
Watch On

Ever wondered what AI could be used for in games? We sure have. A lot. There are hundreds, if not thousands, of possible use cases for machine learning and artificial intelligence in game development. Yet despite letting our imaginations run wild, we are still blown away by what a team of researchers from Nvidia have achieved today: a functioning AI that can replicate Pac-Man with nothing more than pixels and key presses.

The generative adversarial network (GAN) outlined in the research paper, nicknamed GameGAN, is capable of taking pixel and input data from a videogame and recreating a like-for-like carbon copy. It does so without an underlying engine—the AI actually generates a new frame for every on-screen event based on those before it, player action, and a hint of environmental randomness.

Nvidia GameGAN Pac-Man

(Image credit: Nvidia)

Its first task was building the videogame classic Pac-Man—today marks the videogame's 40th anniversary. Don't be deceived by its simplistic appearance, either. Even a game such as Pac-Man, which is a far-cry from the complexity of today's AAA titles, makes for a daunting task. That's because in order to build the game the AI must first understand its rules.

To do so, it needs gameplay data. Lots of it. Nvidia went about building a second AI that could play Pac-Man over and over again to collect enough of the stuff. It took 50,000 'episodes' of gameplay (a few million frames in total) to feed the GameGAN with enough information to replicate the underlying game logic.

That's only basic information, too. GameGAN learns with just the on-screen pixel information and the key presses registered during gameplay. None of the underlying code required to build the game or run it is passed on to the AI.

From there, GameGAN was taught (across a couple of Nvidia DGX systems) to produce a ruleset for its own imitation game. For example, it must learn that walls block Pac-Man's movement in order to keep players from breaking the game at every turn. 

Three main modules within the neural network help realise Pac-Man:

  • The dynamics engine has access to past game history, and uses that wealth of information to ensure a consistent simulation that matches the source game.
  • The memory module keeps track of the environment and maintains consistency, in this case with the Pac-Man environment and background.
  • The rendering engine is tasked with actually producing the output image. It bases this on all the information fed to it by the dynamics engine, and with the static and consistent information kept in line by the memory module. It also uses a specialised rendering engine architecture to produce disentangled scenes, which can also be used to switch out static and dynamic components. This can then be used to switch out characters and/or backgrounds at will.

The resulting game from the fully-trained model is a Pac-Man experience not unlike the real thing. It also delivered another game mode called "ego-centric", in which Pac-Man takes centre-stage and moves about the map with limited visibility on what's ahead.

"We think this is the seed of many different uses of AI-assisted game building." says Rev Lebaredian, VP of simulation technology at Nvidia, in a pre-briefing call. "Before you do any of those things you need an AI smart enough to understand basic rules. Once it understands rules, you can use the AI to do other things, such as construct levels."

GameGAN can even produce new levels and environments using existing information, and the researchers even believe it could one day serve to help develop and simplify developer- and user-generated levels and characters.

"Not just the game developers can be helped, but also the end users" says Sanja Fidler, director of AI at Nvidia. "...the neural network, in order to learn to recreate the game, has to separate components and has to learn about those different components. One could imagine the end user actually modifying the game in this way."

The research team behind GameGAN hails from Nvidia, the University of Toronto, Vector Institute, and MIT. The full research paper can be found here. Their research joins the ranks of other Nvidia-led projects such as GauGAN, the photorealistic image generation app; Image Inpainting, a photo reconstruction tool; and GANimal, a pet face-swapping app. Yeah, AI can be pretty weird.

What's next for GameGAN, you ask? Nvidia's researchers haven't ruled out 3D videogames just yet. "There's nothing fundamentally limiting with this technology to 2D," we're told. And its remit also goes far beyond just videogames. The researchers also believe an advanced AI capable of learning virtual rulesets could be important to development of AI with real-world applications and would minimise the work involved with generating a true-to-life simulation.

The possibilities are near-enough endless for game development and AI, and they feel like a perfect fit for one another. GameGAN feels like just the beginning of AI's relationship with videogames. It's an exciting (and only a little daunting) glimpse of what's going to be possible with more complex and advanced AI algorithms in the future.

If you'd like to try out GameGAN for yourself, you will be able to do so from later this year on Nvidia's AI Playground. That's also where you'll find the other AI-powered apps mentioned above, which are all ready to try out whenever you're ready.

TOPICS
Jacob Ridley
Managing Editor, Hardware

Jacob earned his first byline writing for his own tech blog. From there, he graduated to professionally breaking things as hardware writer at PCGamesN, and would go on to run the team as hardware editor. He joined PC Gamer's top staff as senior hardware editor before becoming managing editor of the hardware team, and you'll now find him reporting on the latest developments in the technology and gaming industries and testing the newest PC components.

Latest in AI
Otter AI Meeting Agent
As if your work meetings weren't already fun enough, now Otter has a new all-hearing AI agent that remembers everything anyone has said and can join in the discussion
Image for
'No real human would go four links deep into a maze of AI-generated nonsense': Cloudflare's AI Labyrinth uses decoy pages to trap web-crawling bots and feed them slop 'as a defensive weapon'
CHINA - 2025/02/11: In this photo illustration, a Roblox logo is seen displayed on the screen of a smartphone. (Photo Illustration by Sheldon Cooper/SOPA Images/LightRocket via Getty Images)
'Humans still surpass machines': Roblox has been using a machine learning voice chat moderation system for a year, but in some cases you just can't beat real people
OpenAI logo displayed on a phone screen and ChatGPT website displayed on a laptop screen are seen in this illustration photo taken in Krakow, Poland on December 5, 2022.
ChatGPT faces legal complaint after a user inputted their own name and found it accused them of made-up crimes
Public Eye trailer still - dead-eyed police officer sitting for an interview
I'm creeped out by this trailer for a generative AI game about people using an AI-powered app to solve violent crimes in the year 2028 that somehow isn't a cautionary tale
Closeup of the new Copilot key coming to Windows 11 PC keyboards
Microsoft co-authored paper suggests the regular use of gen-AI can leave users with a 'diminished skill for independent problem-solving' and at least one AI model seems to agree
Latest in News
The heroes are attacked by monsters
Pillars of Eternity is getting turn-based combat to mark its 10th anniversary, and that means PC Gamer editors will soon be arguing about combat mechanics again
Image of Ronaldo from Fatal Fury: City of the Wolves trailer
It doesn't really make sense that soccer star Ronaldo is now a Fatal Fury character, but if you follow the money you can see how it happened
Junah beginning a battle in Metaphor: ReFantazio.
Today's RPG fans are 'very sensitive to feeling like they wasted time' when they die, says Metaphor: ReFantazio battle planner—but Atlus still made combat hard anyway
Image of Cersei Lanniser from Game of Thrones: Kingsroad Steam early access trailer
A new Game of Thrones RPG is coming to Steam today with a cast of 'familiar faces,' which is good because it's really the only way to tell it's a GoT game at all
The new Prime Asset featured in the upcoming update for the Outlast Trials.
The Outlast Trials puts its already paranoid players under surveillance for a time-limited story event
A Viera looking confused in Final Fantasy 14.
Old armor continues to fall victim to Final Fantasy 14's bizarre two-channel dye system, unless you're super into changing the colour of teeny-tiny eyelets: 'Why even bother at this point?'