For as long as I can remember, photorealism has been a highly sought after milestone for gaming. I guess it's just one of those things that us humans feel like we need to achieve. The latest attempt to do just that comes from the Intel Intelligent Systems Lab, where a team of researchers have built an AI that will take a game's art style and remix it into something approaching photorealism.
It's called the "Enhancing Photorealism Enhancement" and it uses a convolutional neural network to analyse and alter a frame to make it appear more true to life.
The perfect test case? GTA 5, according to researchers Stephan R. Richter, Hassan Abu AlHaija, and Vladlen Koltun. Check out the results in the video below.
It's a bit of a mixed bag. From a gamer's perspective, it looks grey, green, and a bit drab. It gives me early-2000s FPS vibes—like the saturation has been sucked right out of it. From a research perspective, however, it's clearly a huge success, and I think that's really what to focus on here.
Best gaming motherboard: the best boards around
Best AMD motherboard: your new Ryzen's new home
The ML technique is a way of transplanting photorealism onto an image that hasn't been rendered as such. To do that, you'd need fully ray-traced scenes and highly-intensive workloads the likes of which only major animation studios can cope with frame-by-frame.
The researchers are also able to mix up the results through the use of different training datasets. The 'cityscapes' dataset shown in the video may cause a rather drab look, but the scene enhanced with the 'Mapillary Vistas' dataset is far brighter and more colourful.
Clearly there's more to be done to get this technology off the ground and actually useful for gamers, or indeed any form of media. The Intel Labs researchers believe that its enhancement could be easily transplanted into game engines for deeper, more realistic results, which could open up a whole new world of in-game customization, if done right.
The process will need to be sped up through optimisation before we get anything close to playable frametimes, though. Inference with the Enhancing Photorealism Enhancement today takes half a second on an RTX 3090. Ouch.
