Zach Barth, the creator of hyper-clever puzzle games Opus Magnum and SpaceChem, bumped into David Galindo, maker of Cook, Serve, Delicious!, in which you cook meals rapidfire and run a restaurant, at PAX. Galindo had a problem. He'd been preparing to port Cook, Serve, Delicious! 2!! to consoles, and he'd just realised that one of the challenges he'd set in the PC version a year earlier was impossible to complete.
Metrics just blow the doors wide open to understanding your own game.
David Galindo
The challenge was to make all orders on every level perfectly. But Cook, Serve, Delicious! 2!! has 400 levels, and, being a lone developer, Galindo simply didn't have the resources to test and balance them all. So he'd relied on making the levels fair with math, setting time limits according to what food was being cooked.
He never heard any complaints, so Galindo assumed his system had worked. But players simply weren't getting that far in the game. Other than through Steam's public (and very basic) achievement data, reviews and streamers, Galindo had no way of understanding how people actually played his game, how far they got, and why they stopped playing.
"I was so flabbergasted. Oh my god," says Barth. That's not the way he makes games.
Before he was a game developer, Barth worked at Microsoft on Office, where software engineering came with metrics as a matter of course. Not to surveil or extract more money from users, but to know what they do and don't use. All of Barth's games collect anonymous metrics on how they're played. And here was a developer friend telling him that he had no idea he'd made a part of his game impossible.
"I never included them until now because I'm the sole programmer for my games, and I'm not familiar with networking and how to properly implement metrics and gathering anonymous data, and because quite honestly I didn't realise how absolutely crucial it can be to game making," says Galindo. "I didn't realise how blind I was until I saw the way Zach implemented his metrics in Opus Magnum, and it just blows the door wide open to understanding your own game."
Barth can see how many players reached every one of the levels in his games, and how many succeeded. He can see how long they took and how many got stuck and never come back. He can see where the difficulty spikes are and do something about them, and he can take what he learns from one game and apply it to the next. As the lead of a very small studio making, frankly, niche games, metrics mean he doesn't need to spend enormous amounts of time and money he doesn't have on testers and marketing, and instead spend it on fascinating, clever game design.
Metrics are vital to modern game development. Most mainstream PC games quietly collect them, and while it can feel disconcerting to know that your PC is feeding information about how you play to a server somewhere, it's a big part of the reason games are so complex, and so able to change and improve over time.
What data do developers take?
When you start Divinity: Original Sin 2 for the first time, you'll see a message pop up saying "Support Us With Your Feedback." It's a front for a large system that developer Larian use to understand how their sprawling RPG is played, and it informs you that if you agree, the game will automatically send Larian anonymous details about your system and 'gameplay session files' whenever you finish playing.
What does Larian gather? Well, that's changed over time.
"During Steam Early Access we experimented a bit with what we could gather, and we logged almost every single choice, action and footstep," producer David Walgrave says. But that was kind of a mistake. The sheer bulk of data clogged up Larian's servers, slowed down the game as it gathered and transmitted it.
This wall of data was useful, however, for teaching Larian something important. "Logging is not about data, but about asking the right questions first," Walgrave says. Questions like: where do characters die? Where on the map do they level up? What equipment do they have when they level up? What is the most popular weapon? How much gold do they carry when they enter and leave a region? What talents are being selected?
"These types of questions are used to try and get a good picture of balancing and gameplay. For instance, if a skill is not being used, or seldom being used, there could be several things going wrong, and we try to find out what exactly. Is the skillbook not available? Is it too expensive? Do people think the skill is not powerful enough or useless?"
It's all down to asking the right questions of the data. And that's all down to someone writing the right database query, which delves into gigabytes of unreadable numbers and comes back with pie charts, graphs and heatmaps. "It's pretty cool and magical, obviously, and we all sit around smoking cigars and pointing at the charts, nodding wisely," says Walgrave.
Fatshark, developers of the Vermintide series, ask very similar questions, despite making a very different kind of game. "If a certain weapon type is the most popular, we can focus on making more of that particular weapon type," says Fatshark's chief operating officer, Sven Folkesson. "Or, we might see that comparatively few are playing a certain level, so we might want to make changes to it or explain the missions better."
There might be a group of players that are very vocal about a certain thing in the game, and it's easy for a developer to focus on that very thing.
Sven Folkesson
The answers help Fatshark prioritise because they show how the game is actually played. Vermintide has forums where players continually report issues, and Fatshark can also monitor comments on Reddit, Discord and beyond, but they know that these sources only give them a partial view of what players think and want.
"Not everybody will post about their experiences,' says Folkesson. "There might be a group of players that are very vocal about a certain thing in the game, and it's easy for a developer to focus on that very thing." Data can confirm whether reported issues go beyond super-engaged players or whether subtler problems are worse, and that helps Fatshark figure out how to divide up development resources.
At one point data showed that a Vermintide 2 level had a much lower completion rate than the rest, so the developers immediately started looking for the reasons. "We found out that the level had far too many roamers early on, in addition to a boss spawn, so we removed a few roamers to make the level on par with the rest."
Today's multiplayer games are constantly in flux, and use data as well as player feedback to inform balance patches. "We have data that tells us how often a champion will win a game depending on if they’re in the top, mid, the jungle, or in a duo bot lane," wrote one of Riot Games' developers in a blog post three years ago. Valve collects tons of data from CS:GO matches, and uses that data to tweak map design. Blizzard's developers on Overwatch and Hearthstone often talk about how they balance games based on a combination of feedback and stats—both are crucial to getting a complete picture.
When Larian came to review all the character skills for D:OS2's Definitive Edition, the data showed that hardly anyone was using the Summoner class' Planar Gateway, which sets two gates down in the world so characters can travel between them. Nervous that players would use it to mess up combat by teleporting characters out of fights, Larian originally made it cost a prohibitively expensive six action points. In Definitive Edition it's free, but with a limited number of jumps.
Metrics-based design is not sexy or necessarily filled with eureka moments of genius. Metrics instead drive a quieter but essential side of game development, in which details that no one notices are tweaked to make things consistently fair and fun.
"And it also helps us as a company," says Larian's Walgrave. "We gain insight into how to do things better, and we see what is popular, what works, and we can try to understand that and take better questions to the players next time we see them."
The privacy question
The question that looms over the whole topic of data collection is, of course, privacy, particularly in a post-GDPR, post-Cambridge Analytica world. All the developers we spoke to only deal in anonymised data and use it to look for general trends, not specific granular details, and they take privacy very seriously.
And as well as play metrics, games also collect other forms of data. You may remember that in June news broke that a service called Red Shell was present in a large number of popular games, and that it "tracks data of your PC and shares it with 3rd parties." Amid such words as 'spyware' and 'data mining software,' the furor led to many developers hurriedly removing Red Shell from their games.
Fatshark was one of them. "If our players are concerned, we have to take that seriously," Folkesson tells us. "We are not in the data business, we're in the games business, and our games and our players are our be-all and end-all."
But there was a reason Fatshark, along with Firaxis Games, Creative Assembly, ZeniMax Online Studios, and many other leading developers used Red Shell. It's a 'marketing attribution' service, which gives developers an idea of how effective their marketing marketing campaigns are.
"Basically, we could see that a certain percentage of people who clicked on a certain marketing campaign link then at some later date had launched the game," says Folkesson.
It's always shocking to learn that information about us is being quietly collected, but Red Shell isn't quite the invasion of privacy that it might have seemed. Marketing attribution is used across the web, as well as in smartphone apps and also many non-Steam PC games, from League of Legends to World of Tanks. Red Shell is merely the first service to support Steam games.
The problem, of course, is that no game ever clearly stated that data was being collected, no matter if it was anonymous.
Adam Lieb, founder of Red Shell maker Innervate, sees it like this: until Red Shell, developers of games for Steam have been at a disadvantage because Steam makes it hard to gather attribution data. And, he says, that difficulty is a big reason why various big-name games aren't being released on Steam: "Now that you see people like Bethesda choosing not to launch Fallout 76 on Steam and instead on their own platform, it becomes clear that it's a huge advantage." (That decision may have more to do with keeping the extra 30 percent of sales that Valve and other stores commonly take, but data likely does play a role).
"The platforms that sell our games, or any other e-commerce platform, for that matter, have great insights into their customers behaviour, but they are not sharing it with us," agrees Fatshark's Folkesson. Just like player data allows Fatshark to allocate their development resources, Red Shell enabled them to see how best to spend their limited marketing budget.
Red Shell works by connecting two pieces of information: a specific ad campaign seen in a web browser and an installed game. For the first, Red Shell creates a 'fingerprint' for everyone who sees an ad, which comprises the same data about your computer that any website collects: your screen resolution, IP address, your browser. It places a cookie to avoid tracking the same person twice. For the second piece of information, a Red Shell-supported game sends to Red Shell another fingerprint containing the same data: IP, resolution and so on. It doesn't install anything, it's part of the game itself, and it doesn't gather sensitive data from across a hard drive.
Then, on its servers, Red Shell compares fingerprints from these two sources. When they match, it assumes that it's evidence of a person who installed the game and also saw the ad campaign. Developers only get to see statistics on these matches: no personal or identifying info is collected.
The problem, of course, is that no game ever clearly stated that data was being collected, no matter if it was anonymous. Things blew up and, says Folkesson, "We are back to not knowing how our limited marketing spend is actually driving sales." As an independent developer making complex, high quality games, that's a challenge. Fatshark is a little less efficient and its marketing team needs slightly bigger budgets, taking money which could have been spent on making games.
As gamers become more aware of data collection through new legislation like GDPR, and as new scandals blow up, it will become ever more important that developers take Larian's course and issue a friendly opt-in screen when launching a game.
"Personally I have always been a fan of correct and clear treatment of user information," says Walgrave. "I mean, nobody likes spam or spyware or uploads you don't know about. So we are being very transparent to our players, and tell them everything truthfully." After all, metrics aren't going away. And, frankly, games are better for them.
As for Galindo, after learning all about Zachtronics' data gathering, he's very happy. He's just about put a metrics system in Cook, Serve, Delicious! 2!! that Barth made for him after PAX.
"I'm super excited to finally see just how exactly people play my game. I honestly can't thank him enough. It's going to be a staggering change to my game making once I have it implemented in my game, thanks to Zach. He's the best!"
