Modders are using AI to upscale pre-rendered PS1 backgrounds with phenomenal results
This may be the closest we ever get to "HD" remasters of early 3D games. Here's what ESRGAN is, and why it's so exciting.
PC modders have a history of fixing and improving ports of Japanese games, and the latest case is one of the coolest, most exciting applications of technology to old games I've ever seen. We recently wrote about how some people are using neural networks to upscale the backgrounds of Final Fantasy 7, and in just a few weeks these projects have made enormous gains in quality and complexity, and for more than just Final Fantasy 7. Anyone can use these tools, and as a result we're at the start of what feels like a renaissance for PlayStation-era 2D art.
A little background: When 3D was in its infancy, PlayStation game developers compensated for the hardware's shortcomings by pairing 3D character models with pre-rendered 2D backgrounds. Many of them, from series like Final Fantasy, are still stunning today: Detailed, stylized, beautiful works of art you could walk through as you went on an adventure. But the art files on those discs were naturally low resolution, and sloppy archival meant that when Square Enix finally ported games like Final Fantasy 8 and 9 to PC in "HD" form, it didn't have the original art files to use for modern high-res displays. Square simply upscaled the low-res files from the original games to 1080p and called it a day.
The results weren't great. The backgrounds can be blurry or oversharpened—just like if you opened a jpeg in Photoshop and tried to blow it up to four times the original size. But what if ~the magic of AI~ could analyze the image and more intelligently resize it? That's what ESRGAN, or Enhanced Super Resolution Generative Adversarial Networks, are all about. OK, that name is a mouthful, and I'm not about to pretend I fully understand how it works. But there's a whole field of study around the concept of perceptual super resolution, and competing AI implementations for how to upscale images with the best results. ESRGAN is one of the most successful.
Game fans have latched onto this technology and are dialing in how to best use it for low-resolution background images. Here's a blog that walks you through how to install ESRGAN on Windows and shows off a few results. This forum thread compiles even more examples of the AI applied to different game backgrounds and textures.
The most exciting project to me right now is for Final Fantasy 9, one of my favorite games and one with phenomenal art that doesn't look too hot upscaled to 1080p. Modder Ze PilOt is using this technique to upscale FF9's backgrounds and other textures, which is much more involved than upscaling a jpeg:
So, this nice little tool allows to extract background from the Unity asset files & re-inject them. https://t.co/uBeBQry8Fg. Inside the engine, a background looks like this. pic.twitter.com/5KQLnKt8pJDecember 30, 2018
As he tweeted in December, "every frame of the animation is on a different layer. Lighting is mixed with the background. I need to separate all these frames into single images."
But he's made huge progress already, and is rapidly iterating on a mod implementing these new backgrounds and making other touches, like throwing in the old PS1 font to replace the PC version's bland new one. You can grab version 4 of the mod, Moguri Memoria, here, and follow its development in this forum thread.
The biggest gaming news, reviews and hardware deals
Keep up to date with the most important stories and the best deals, as picked by the PC Gamer team.
Here's a great example of an original texture next to an AI-upscaled one:
And now, ingame assets & battle fields... pic.twitter.com/yL3kQQxXiFJanuary 25, 2019
These upscales will, of course, never be as good as having the original art assets in high resolution. They'll be blurrier in places and lose detail in others. Purists can argue they distort the original artistic intent, and a poor implementation definitely can. But from what I've seen, these AI-driven upscales are vastly better than the pixelated versions we've seen in some official releases, and could be the best way to modernize games with pre-rendered backgrounds for modern displays.
For now, the easiest targets for these upscales will be the PC ports of older games, but expect to see fans also upscaling backgrounds that can then be injected into games via emulators like Dolphin for the GameCube/Wii and PCSX2 for the PS2. For a long time, emulators have been able to greatly enhance 3D graphics but rendering them in higher resolutions—we're just beginning to see how technology can help us do the same with 2D graphics without destroying the style of the original artwork.
Wes has been covering games and hardware for more than 10 years, first at tech sites like The Wirecutter and Tested before joining the PC Gamer team in 2014. Wes plays a little bit of everything, but he'll always jump at the chance to cover emulation and Japanese games.
When he's not obsessively optimizing and re-optimizing a tangle of conveyor belts in Satisfactory (it's really becoming a problem), he's probably playing a 20-year-old Final Fantasy or some opaque ASCII roguelike. With a focus on writing and editing features, he seeks out personal stories and in-depth histories from the corners of PC gaming and its niche communities. 50% pizza by volume (deep dish, to be specific).
'I want to incorporate more changes, I don't want to take a conservative approach': Final Fantasy 14 game director Yoshi-P expresses a desire to shake things up in the future
Final Fantasy 14 director Yoshi-P talks about the Scion's 'half-hearted' role in Dawntrail, says they'll likely use smaller, more selective groups of beloved NPCs in the future