Queen's University analysts Dayi Lin, Cor-Paul Bezemer, Ying Zou and Ahmed Hassan recently published an empirical study of Steam reviews. The study focuses on how user reviews of games on Steam compare to reviews of mobile apps and games, but its findings say plenty about Steam itself and are pretty darn interesting.
Here's how it went down. Lin and team took a snapshot of the 8,025 games available on Steam in March 2016, trimmed that down to 6,224 games by removing games with less than 25 reviews, and ran the 10,954,956 reviews for those games through a mix of automatic and manual analyses (mostly automatic given the volume of reviews). They also relied on user data from Steam Spy (a service that was nearly killed by a recent Steam privacy update, making the timing of the study fortuitous).
Players complain more about game design than bugs
Lin et al.
The numbers say the average Steam review is about 30 words long and sits at an eighth grade reading level (that is, according to the Coleman-Liau readability index used by the US Office of Education). The study found that reviews for Early Access games were slightly longer, and that free-to-play games received shorter reviews than paid games.
Like Early Access games, indie games (that is, games tagged 'indie' on Steam) received longer reviews, but uniquely their reviews contained more design suggestions. In other words, reviews of indie games included more comments like "I wish it had this feature" or "It might be better if it did this." Lin and his team said that a possible explanation for this uptick in suggestions is "that the player community of indie games is more engaged than the community of non-indie games." It's also possible players feel their feedback is more likely to be taken seriously when speaking with smaller teams of devs, though that is our own speculation.
The study estimates that 42 percent of Steam reviews provide "valuable" feedback, value defined here as specific pros, cons, or bug reports. Although the study found that more of that sort of feedback appears in negative reviews, Lin and the team stress that positive reviews can also provide useful information, such as which features are important to players and can be expanded. The actual 'value' of a review will vary, of course, as it may not tell a developer anything new about their game or audience, even if it does contain specific complaints or suggestions.
Developer and analyst Lars Doucet, who helps conduct an annual survey on what developers think of Steam, offered support to this claim, tweeting that negative reviews are often the most helpful in his experience. For more on Doucet's work, check out his write-up of last year's Steam survey.
"Players complain more about game design than bugs," the study says. Only eight percent of reviews mention bugs, while 34 percent mention "bad" or annoying game design. This was even truer of negative reviews: 17 percent of negative reviews mention bugs whereas 57 percent mention game design. Moreover, negative reviews mentioned bugs like game-breaking freezes or crashes, whereas positive reviews mostly mentioned charming or inoffensive bugs like wonky textures or the occasional FPS dip. Bethesda can rest easy.
Indie game developers have a shorter time to satisfy players
Lin et al.
Lin and the team also found that while most reviews are written after around 13 hours of play, "the majority of negative reviews" are posted within the first seven hours. Additionally, free-to-play games see a spike in reviews after just one hour of play.
Here's a fun fact to help cut through the numbers: Stardew Valley saw the highest review volume in the period the study examined, at a staggering 586 reviews per day. To put that into perspective, the study found that 96 percent of games receive less than 10 reviews a day. CS:GO received 514 reviews per day in the same period.
Stardew's hotness is unsurprising, but it's especially relevant considering indie games also stood out in the category of review timelines. "Indie game developers have a shorter time to satisfy players in their games than non-indie game developers," the study reads. This is just me spitballing here, but I reckon that has something to do with the volume of indie games on Steam and their generally lower prices.
