Gaming performance analysis 101

Last month, we launched a new way of doing performance analysis on game releases with Mass Effect: Andromeda. There was a lot of ground to cover and not enough time to go over all the various bits of hardware we're testing, or the arrangements with MSI, who provided all the hardware. This is my chance to properly introduce what we're doing, why things are MSI branded, and the plans for the next year of performance analysis articles.

The first thing to get out of the way is that these articles are in addition to the regular GPU and CPU reviews I do, and new graphics card launches will continue to be done in the same manner as before—eg, see the GTX 1080 Ti review. This also isn't a strictly new thing we're doing, as I've previously looked at games like Battlefield 1, Fallout 4, Rise of the Tomb Raider, Ashes of the Singularity (with explicit multi-adapter support), Doom (and later the Vulkan API update), Gears of War 4, Mafia 3, and more.

Doing full blown benchmarking articles for new games takes a lot of time and a lot of hardware. MSI agreed to partner with us to provide both the hardware used for testing (more on that in a moment), and some funds to offset the cost of me being buried neck deep in testing for a week.

It's not just about my testing and writing, as our video team will be putting together content as well. I helped create some tools that provide us with real-time overlays of framerates, and you can see the end result with our Mass Effect: Andromeda performance analysis video. It turns out creating framerate overlays for actual video content is tricky, but I hacked together a solution and I hope you find it useful. I'll be looking at improving the output for our next video, but we're off to a good start.

MSI for its part gets its logo on the charts and video. And really, that's about the end of the sponsorship. I'm free to choose the benchmark sequence, the games I test, and I can say whatever I want about the various GPUs.

Meet the graphics cards

Let's talk about the hardware that MSI sent for testing. First, we have the complete set of modern graphics cards—okay, not the complete set, as I didn't request 4GB RX 480, 8GB RX 470, or 4GB RX 460 cards, but I've got all the major cards from the budget-friendly RX 460 and GTX 1050 up through the GTX 1080 Ti—and once RX Vega arrives, I'll get that as well.

MSI sent its Gaming X variant of every card (except the RX 460 2GB), which means dual fans and big coolers, with a red and black motif. These cards are all factory overclocked for added performance, but they're also tuned for quiet computing, so the fans will actually stop spinning if temperatures are below a certain threshold. Since most gamers tend to prefer custom cooling solution rather than the reference designs, MSI's Gaming X GPUs represent more of a real-world view of the market (albeit at a price premium).

All of these are newer graphics cards, and people often want to see what performance looks like on older hardware. If there's time (and the game doesn't lock me out for 24 hours after five hardware configurations), I'll see about including a couple of popular previous generation GPUs like the R9 390 and GTX 970. Otherwise, in practice the GTX 1060 6GB is typically a bit slower than a GTX 980, while the GTX 1060 3GB is a bit faster than a GTX 970. The R9 390 tends to split the difference, landing pretty close to the 1060 6GB on average, but it has games where it's quite a bit faster and others where it's quite a bit slower.

Desktops and notebooks for testing

Along with the graphics cards, MSI sent an Aegis Ti3 desktop PC. This is a custom design that uses a proprietary motherboard layout and a funky case, with the GPUs sitting at the top of the case and mounted vertically rather than horizontally. The Aegis Ti3 also supports SLI and CrossFire, and the model we're using ships standard with a pair of GTX 1080 cards. It also includes an overclocked 4.8GHz i7-7700K, dual Plextor M8Pe 512GB M.2 NVMe SSDs in RAID0, and 64GB of DDR4-2400 memory.

Something I'll also be testing in a more limited fashion is Ryzen 7 gaming performance, using MSI's X370 XPower Gaming Titanium motherboard. I've equipped that system with a non-NVMe SSD for now, plus 16GB DDR4-3200 memory, and a Ryzen 1700 running at 3.9GHz. Usually I'll check performance with the fastest GPU, just to see how much of a difference there is between Intel and AMD platforms, but I might include one or two midrange options as well.

Flipping over to the mobile side of things, we have three MSI gaming notebooks, the GT73VR, GT62VR, and GS63VR. These use the GTX 1080, GTX 1070, and GTX 1060 6GB, respectively, with the GT73VR running an overclockable i7-7820HK while the other two use an i7-7700HQ. The GT73VR also has the GTX 1080 overclocked by default, which results in performance that's actually higher than the desktop 1080.

This isn't a full review, but these are great gaming notebooks if you're looking for that sort of thing. They check all the right feature boxes, and more importantly there are no critical flaws—like a crappy display, mushy keyboard, cheap feeling chassis, or a touchpad that doesn't work right.

All three include M.2 NVMe storage, plus a hard drive for holding larger games and other bulk data. The GT73VR and GT62VR use 120Hz G-Sync displays while the GS63VR has a 4K display. Personally, I'd give up the 4K and never really miss it on a 15.6-inch laptop, but you can configure the display as you see fit. There's also the requisite customizable RGB lighting on the keyboards, something MSI has been doing as long as anyone in the notebook space.

How we test

The process of testing gaming performance varies by game. Some games have a built-in benchmark that makes testing easier, but I also like to verify that any built-in test actually represents real gameplay—so a strategy game played from a top-down view with a benchmark that focuses on in-engine cutscenes probably isn't ideal.

If there's no built-in test, I play through the game for a bit to find a good test sequence. But what makes for a good test sequence? Personally, I shoot for areas that tax the hardware more rather than less, so in the case of Mass Effect: Andromeda, I used an outdoor area. The rationale is that gamers are more worried about what happens in demanding areas where framerates can stutter—if you're getting 150 fps, you would normally increase the quality settings or resolution.

Whether or not a game includes a built-in benchmark, I normally log frametimes using utilities like FRAPS or OCAT (which is based on PresentMon). This allows me to report the average framerate, as well as the 97 percentile framerate. This latter requires a short explanation.

Minimum framerates are important, but a single slow frame in a test sequence is far less important than consistently slow frame. For the 97 percentile results, I first calculate the 97 percentile—the frametime that is longer than 97 percent of frametimes—and then I calculate the average framerate of all frametimes higher than this value. This provides a realistic view of the minimum fps, and if it's a lot lower than the average fps it's a good indication that there's lots of judder/stuttering present. The video framerate overlays will provide a better view of what's actually happening, for those who are interested in that sort of thing.

Finally, for test settings I stick with typical maximum quality for 1080p and 1440p, with the option to downgrade to high/very high if that proves too demanding. At 4K, I'll test either high or ultra, again depending on how taxing the game is. And finally, to show what budget-oriented hardware can do, I test at 1080p medium quality.

To be continued…

So where do we go from here? I'll be doing benchmarks, including videos, for at least twelve games during the coming year. That's twelve more videos than last year, and three times as many games. And this will all be in addition to our normal graphics card reviews, which may incorporate some of the games that I test. A big thanks to MSI for providing the hardware and support to make these articles happen.

We have a list of games we're planning to give the full performance analysis treatment this year. My main concern is that the testing be relevant and useful—so if a game has a 60 fps framerate cap, there's a good chance it won't make for a good performance analysis article. (Framerate caps must die!) The goal is to have the performance analysis done in time to go live with our full review of each game.

In the meantime, if there are any particular games that you'd like to see tested, let me know in the comments. I can't promise I'll test everything, but if it's popular and reasonably demanding in terms of hardware, I'll give it a shot.