Solving bugs in game development is complicated. Solving bugs in game development when your game is actually half a dozen games, ported to PC from three different consoles across two decades, is very complicated. That's the broad thrust of the latest Halo: Master Chief Collection community update, which goes into rare, extensive detail about the process of bug fixing a big game. It's an interesting read that's surprisingly easy to understand even if you aren't a game developer, and serves as a direct response to the unfortunately nasty multiplayer bug Halo 2 launched with.
"These releases have been, and will continue to be, a learning experience for our team," writes senior producer Michael Fahrny. "MCC is a BROAD product and it can be very difficult to make sure everything is 100% where it should be—sometimes things will be missed. Our promise to you is that we will react to situations like these the quickest we possibly can. We are constantly evolving our process to take in new ways to be more efficient with our development and to give our test team confidence that they are getting the time they need to ensure proper coverage is happening."
So what happened with Halo 2? The post first goes through the general steps of finding and fixing a bug, and how every department in a big game studio plays a role. That includes the community team, who interact with players and gather information about reported issues, a dedicated support team that takes reported issues and analyzes them, quality assurance testers, design, production, and engineering. It takes a whole lot of people and a remarkably complex process to nail down a bug and fix it.
Halo 2's PC multiplayer featured a bug that hadn't cropped up on the Xbox, which could cause teleporting projectiles and instant death. It was bad news. And if you distill the cause down to the most basic explanation, all that havoc was caused by a single number: -1.
Here's the more complex explanation, if you can follow it:
"The issue was related to interpolation code that was added in fairly late and is only compiled in PC builds, and unintentionally the dedicated server build. Xbox builds do not compile the interpolation code. One change for interpolation was in a function that gets the camera position of a biped. Players control bipeds. At most, there are four local players, or ‘users’. This function was tweaked so that it would update and read from the interpolation data for the user controlling that biped. However, it was not handling the case where the user controlling the biped did not exist on that machine. So, the user index for non-local bipeds would be -1, which then caused entirely different memory in the interpolation state to be written to and read from. Well, this is further complicated by the fact that this same code path is used by the function that creates a weapon’s projectiles. Because to do that, the game needs to know where the player’s camera was aiming. When it comes to the host and remote players, the same code path is still taken but using the predicted weapon fire state of the remote player. Not all weapons are fired in the same predictive manner, which is why some weapons would inflict damage, while other weapons would not.
"The fix here was one, not compile the code for the dedicated server, but also two, guard against cases where the user index was -1."
Unrelated to this particular bug, there's some cool development insight from engineer Sean Cooper about the process of restructuring Halo 2's network code to work on PC, where different speed hard drives and SSDs would otherwise cause timeout issues.
But onto the new stuff: Halo 3 testing should start "within the first few weeks of June," and will include Forge and Theater, as well five campaign missions and plenty of multiplayer maps and playlists. There's also some movement on other games in the MCC, including the addition of its Challenges system, audio fixes for Halo Reach, bug fixes around screen tearing, and secondary key bindings. All those things are getting closer to release and should be in an upcoming flight or release.
There are also a couple new additions to the "design iteration" stage that get me excited, including a custom game browser and PC/Xbox crossplay. Design iteration means they're still a long way off, but any step towards a custom game browser on PC is a step I'm happy to see.
