Ever wondered what AI could be used for in games? We sure have. A lot. There are hundreds, if not thousands, of possible use cases for machine learning and artificial intelligence in game development. Yet despite letting our imaginations run wild, we are still blown away by what a team of researchers from Nvidia have achieved today: a functioning AI that can replicate Pac-Man with nothing more than pixels and key presses.
The generative adversarial network (GAN) outlined in the research paper, nicknamed GameGAN, is capable of taking pixel and input data from a videogame and recreating a like-for-like carbon copy. It does so without an underlying engine—the AI actually generates a new frame for every on-screen event based on those before it, player action, and a hint of environmental randomness.
Its first task was building the videogame classic Pac-Man—today marks the videogame's 40th anniversary. Don't be deceived by its simplistic appearance, either. Even a game such as Pac-Man, which is a far-cry from the complexity of today's AAA titles, makes for a daunting task. That's because in order to build the game the AI must first understand its rules.
To do so, it needs gameplay data. Lots of it. Nvidia went about building a second AI that could play Pac-Man over and over again to collect enough of the stuff. It took 50,000 'episodes' of gameplay (a few million frames in total) to feed the GameGAN with enough information to replicate the underlying game logic.
That's only basic information, too. GameGAN learns with just the on-screen pixel information and the key presses registered during gameplay. None of the underlying code required to build the game or run it is passed on to the AI.
From there, GameGAN was taught (across a couple of Nvidia DGX systems) to produce a ruleset for its own imitation game. For example, it must learn that walls block Pac-Man's movement in order to keep players from breaking the game at every turn.
Three main modules within the neural network help realise Pac-Man:
- The dynamics engine has access to past game history, and uses that wealth of information to ensure a consistent simulation that matches the source game.
- The memory module keeps track of the environment and maintains consistency, in this case with the Pac-Man environment and background.
- The rendering engine is tasked with actually producing the output image. It bases this on all the information fed to it by the dynamics engine, and with the static and consistent information kept in line by the memory module. It also uses a specialised rendering engine architecture to produce disentangled scenes, which can also be used to switch out static and dynamic components. This can then be used to switch out characters and/or backgrounds at will.
The resulting game from the fully-trained model is a Pac-Man experience not unlike the real thing. It also delivered another game mode called "ego-centric", in which Pac-Man takes centre-stage and moves about the map with limited visibility on what's ahead.
"We think this is the seed of many different uses of AI-assisted game building." says Rev Lebaredian, VP of simulation technology at Nvidia, in a pre-briefing call. "Before you do any of those things you need an AI smart enough to understand basic rules. Once it understands rules, you can use the AI to do other things, such as construct levels."
GameGAN can even produce new levels and environments using existing information, and the researchers even believe it could one day serve to help develop and simplify developer- and user-generated levels and characters.
"Not just the game developers can be helped, but also the end users" says Sanja Fidler, director of AI at Nvidia. "...the neural network, in order to learn to recreate the game, has to separate components and has to learn about those different components. One could imagine the end user actually modifying the game in this way."
The research team behind GameGAN hails from Nvidia, the University of Toronto, Vector Institute, and MIT. The full research paper can be found here. Their research joins the ranks of other Nvidia-led projects such as GauGAN, the photorealistic image generation app; Image Inpainting, a photo reconstruction tool; and GANimal, a pet face-swapping app. Yeah, AI can be pretty weird.
What's next for GameGAN, you ask? Nvidia's researchers haven't ruled out 3D videogames just yet. "There's nothing fundamentally limiting with this technology to 2D," we're told. And its remit also goes far beyond just videogames. The researchers also believe an advanced AI capable of learning virtual rulesets could be important to development of AI with real-world applications and would minimise the work involved with generating a true-to-life simulation.
The possibilities are near-enough endless for game development and AI, and they feel like a perfect fit for one another. GameGAN feels like just the beginning of AI's relationship with videogames. It's an exciting (and only a little daunting) glimpse of what's going to be possible with more complex and advanced AI algorithms in the future.
If you'd like to try out GameGAN for yourself, you will be able to do so from later this year on Nvidia's AI Playground. That's also where you'll find the other AI-powered apps mentioned above, which are all ready to try out whenever you're ready.
