This is Grand Theft Auto as made by AI

Artificial intelligence figures prominently in a wide swath of games, mostly in the way NPCs react to situations. But AI is capable of much more than having a character duck behind a corner during a shootout and, going in a completely different direction, a couple of AI researchers tapped into Nvidia's GameGAN neural network to create GAN Theft Auto—an entirely AI-generated version of Grand Theft Auto V. The result is pretty remarkable.

The playable demo consists of driving down a short stretch of highway in GTA V. From a modern graphics standpoint, it is not what you would expect—the scene is highly pixelated, and even after upsampling the output, there is still a haze, as if playing GTA V in a dream state.

Even so, there are some impressive details visible in the demo, like generating a shadow underneath the car and accurate sunlight reflections in the rear window that change as the vehicle moves around.

So, how was this made possible? GameGAN is special generative adversarial network (or GAN) created by Nvidia, hence the name GAN Theft Auto. As the name sort of implies, GANs are made up of two neural networks that are adversaries, in a sense. One of them is them is a generator. A generator is trained with certain data, for example photos of humans, and then takes what it learns to create fake content that can pass as the real thing.

The second part is a discriminator. This part is tasked with keeping the generator honest, so to speak, by discerning what is real from what is fake.

"GANs can create images that look like photographs of human faces, even though the faces don't belong to any real person," Nvidia explains. "GANs achieve this level of realism by pairing a generator, which learns to produce the target output, with a discriminator, which learns to distinguish true data from the output of the generator. The generator tries to fool the discriminator, and the discriminator tries to keep from being fooled."

It was around this time last year when Nvidia unveiled GameGAN, the first neural network model that mimics a computer engine using a GAN, and showcased it cloning a fully functional version of Pac-Man. Now it's been used to recreate GTA V.

In GAN Theft Auto, the neural network is the actual environment, and you can play in it, YouTuber Harrison Kinsley says (via Engadget). Kinsley and collaborator Daniel Kukiela borrowed a DGX Station A100 from Nvidia for this project, which was outfitted with four A100 Ampere GPUs and a 64-core AMD Epyc processor.

They trained the GAN with a dozen simultaneous feeds of the highway scene and, based on that data, it learned how the car moves and responds to controls. At first, it was tripped up with how to handle boundaries, but eventually figured out what to do if the car, say, hit a barrier on the side of the road.

It's important to note that none of this is human coded—the resulting demo is entirely generated by the GameGAN, from how the world shifts around when the car moves around, to the controls that manipulate the vehicle.

Not everything is perfect. Beyond the low-resolution graphics, the GAN struggled with what to do with collisions into other vehicles. Kinsley describes a situation in which the GAN simply split a police car in two when the main vehicle ran into it head on. But it did get better over time.

This is a possible glimpse into the future of gaming. It is not hard to imagine an entire game being concocted by a GAN, or perhaps portions of a game. That's a bit far off in the distance, though. In the meantime, GAN Theft Auto is available on GitHub.