When Bungie started to work on Destiny back in 2010, it faced a big challenge.
How do you build a game that blurs the line between singleplayer and multiplayer, where players can meet others simply by hitting "play Destiny," without any traditional matchmaking and lobbies? It's an enormously complicated problem, and Bungie solved it—but the solution came with its own share of problems.
In this article, you'll learn how Bungie accomplished its goal of a seamless online world, and why its netcode requires more upload bandwidth than most games, opens some players to DDOS attacks, and leads to hit registration that can feel different for every player you face in the Crucible.
Network model
The developers themselves describe their networking topology as "uniquely complicated," which is not an understatement as the document from their presentation at the Game Developers Conference 2015 shows. The basis is a peer-to-peer system, which you can read about in our beginner's guide to understanding netcode, where all clients (players) directly communicate with each other. There is no dedicated server like we have in CS:GO, Battlefield or Overwatch.
But what makes the networking in Destiny so special and complex is not just that it uses peer-to-peer. It’s that Bungie set out to create a Shared World Shooter, which meant that they had to connect these peer-to-peer ‘bubbles,’ so that you can meet other players on your journey.
For Destiny 2 Bungie continued to use same network model, with one significant change. The physics host is no longer running on a player’s console/PC, but on a dedicated server—which prevents the annoying host migration issue that Destiny 1 suffered from. It's a step up, but Destiny 2 has other problems.
The privacy flaw
A fundamental flaw of any peer-to-peer network model is that it reveals your IP address to every player you meet in PvE or PvP. When we have a look at the connections Destiny 2 established in one of my 4v4 Crucible matches, we can see three Bungie servers located in the USA, one Limelight Server located in the UK, one Blizzard Server located in the UK and the other seven players located in France, Germany and Turkey.
By revealing the IP address of these players, the game makes it very easy to find out the country and city/town where they live. So, I can then block or throttle the traffic between my client and theirs, or I could launch a DDOS attack to affect their online experience—something streamers have to deal with on a regular basis, and has resulted in persistent problems with the relatively high stakes PvP mode Trials of the Nine. Bungie issued a swathe of two-week bans to DDOSers recently, but the attacks keep coming.
Security and hacking
"Never trust the client" is a saying you'll hear often from developers who work on multiplayer games. When you let the client take care of sensitive tasks like hit registration, then this makes it a lot harder to protect your game from hacks.
Bungie faces this problem with the peer-to-peer network model in Destiny 2. To shield the game from hacks on PC, Bungie is preventing applications from injecting code into the game. The downside of this measure is that this also blocks any and all applications that provide an in-game overlay like MSI Afterburner, RTSS or Discord. It also blocks apps used for streaming or gameplay recording like DxTory, and OBS when you try to use the game capture mode. Despite these efforts, which harm legitimate players, hacks still exist for Destiny 2.
Network delay: update rates
The rates at which the game clients send and receive data have a very strong impact on the delay that you experience while playing online. So, when we look at the network traffic from that 4v4 Crucible match that I mentioned earlier, the Limelight and Blizzard servers show next to no activity as these are most likely only used for the user accounts, which is why we will focus on the three Bungie servers.
The first one shows next to no network activity, while the second has quite steady update rates of 10Hz. Based on the information from this dev blog it seems like this server is the Activity Host, responsible for ammo spawn times and the scoring system. The third server has update rates of about 5Hz with occasional spikes. The purpose of this server is not clear, but it’s possible that this is the physics host.
Since Destiny 2 does not use dedicated game servers in the same sense as, for example, Overwatch, the update rates between the clients is the most important factor when it comes to how much delay you experience in this game.
My PC sends about 40 updates per second to all the other 7 players. Sadly I can’t say the same about them, as one of the players from France has an average update rate of just 15Hz which even drops down to 7Hz a few times throughout the match. This can be caused by packet loss, network congestion or when a PC suffers from severe performance issues.
This player's low and inconsistent update rate has quite an impact on my online experience as it increases the lag between us. When you have such a player on the enemy team, you'll notice that it takes your shots quite a bit longer to register when you fire at them. More on that a bit later.
Thankfully this player is the exception, in this match at least, as most of the other players maintain an update rate of around 30Hz. However, there is just one player from Turkey who is able to match my send rate of 40Hz, which is a bit shocking when you consider that lower update rates result in more delay, especially in a peer-to-peer game. The quality of a multiplayer match in Destiny 2 always hinges on the connections of other players.
Network delay: tick rates
Another important factor for any multiplayer game is the tick rate. There has been a lot of confusion and misinformation about whether Destiny 2 runs at a tick rate of 30Hz or 10Hz. Luckily the developers explained in one of their blog articles that Destiny 2 samples combat damage at 30Hz - and not 10Hz. The rumor about the 10Hz probably originated in the fact that the Activity Host sends updates on the scoring and ammo spawn systems at 10Hz.
However it’s not entirely clear if those 30Hz only apply to the console version of Destiny 2, as the PC version sends updates at up to 40Hz.
There are at least 2 possible scenarios:
- The PC version of Destiny 2 samples combat damage at 40Hz not 30Hz
- The PC version of Destiny 2 also samples damage at 30Hz, but the clients send updates more frequently at 40Hz. Battlefield 4 & 1 do the same, which results in a slight delay decrease between client and server.
Network delay: Difference between console and PC
The quality of a player's internet connection affects the rate at which they can send data, but it's not the only factor. Destiny 2's framerate also directly affects how many updates per second a game client can send.
In the images above you can see that Destiny 2 sends updates at 40Hz when its frame rate exceeds 40 FPS, while at 30 FPS the update rate drops to 30Hz. This means that on PC, the maximum update rate of a client is capped at 40Hz, while on console it’s just 30Hz as Destiny 2 is locked at 30 FPS on all consoles, including the PS4 Pro.
Network delay - Hit registration
When you shoot a player in Overwatch, which uses dedicated game servers, it doesn't matter if that player has a ping of 22ms or 522ms. The only thing that matters is your ping to the game server which determines the delay that it takes your shot to register.
In peer-to-peer games you should get lower delays than in games which use dedicated game servers, as you talk directly to the client that you are shooting at. There is no game server between the two players. Unsurprisingly, it doesn't always work out that way.
The lower delays that peer-to-peer can offer fully depend on the game's ability to match with players who live nearby. This theoretical advantage goes out of the window as soon as the game adds players from different countries (like in that 4v4 crucible match that I have been talking about) or even from different continents.
Even worse is that since you have a different ping to each and every other player, the hit registration delay depends on how these game clients reach a consensus on what happened, or what the outcome of an action is—like did this shot hit or miss? In Destiny this leads to kill trades and the infuriating experience of getting shot around corners. It also causes hit registration to feel inconsistent when you fire at players from the enemy team, as you can see in the video below.
Network delay: Test results
Caveat: Since I could not get my two test players into different teams I am not yet able to provide data for the damage delay (yet I expect it to be similar to the gunfire delay). In 2018 Bungie plans to add a private match option to Destiny 2, which will allow me to do that damage delay test then. In this video I explain my testing method in detail:
Since both of my test players/PCs use the same ISP, I get just 1ms ping between them, which means that there is almost no packet transmission delay between these 2 clients.
Under these conditions I measured an average gunfire delay of 36ms and an average movement delay of 37.25ms. At a glance, these delays beat every single game on that list. But there are two things to consider.
1. Ping
If my test players always had a ping of 1ms, then Overwatch would have a gunfire delay of 23ms (13ms less than Destiny 2) and Lawbreakers would have a movement delay of 32ms (5.25ms less than Destiny 2). This also shows that the lower delay that peer-to-peer could provide in theory is not really superior in practice to what you get in games like Overwatch with dedicated servers.
2. Consistency
In Destiny 2 you will experience different delays on a per-player basis, as you could see in the video above. While in games like CS:GO, Overwatch or Battlefield your hit registration will always have the same delay (as long as your connection to the game server is stable).
Bandwidth requirements
While peer-to-peer does not cause any traffic problems in 1v1 fighting or sports games, it’s the worst-case scenario for games which have more than two players, as every additional player increases the bandwidth requirements since all clients talk to each other the whole time.
In numbers, my game client sent 152.3MB and received 128.3MB while playing 4v4 Crucible matches for one hour.
When you compare the upload traffic of Destiny 2 to games which use dedicated game servers, then it becomes clear how much more upstream traffic the peer-2-peer model generates. And that is where a considerable number of players start to struggle, as residential internet connections usually provide far less upstream than downstream bandwidth
Sure, an ADSL connection with 8 Mbit/s down and 1 Mbit/s up should be enough to handle that amount of data. But we're talking about an online game which depends on fast data delivery, and the occasional spikes of nearly 1Mbit/s quickly become a problem. Especially when you consider that the game might not even be able to use your entire up and downstream bandwidth, as it has to share your connection with other applications, services and even other people.
Someone might think now "Why should I care? I have fibre!" Well, peer-to-peer rears its head once again.
With dedicated game servers, your own hit registration is indeed not affected by another player's bad internet connection. But in Destiny 2's peer-to-peer setup, the quality of the players' internet connections, and the distance between them, have a direct impact on how responsive the hit registration feels. And the sad reality is that there are more players who are stuck with an upload bandwidth of 1Mbit/s than you might think.
So the high bandwidth requirements of peer-to-peer do cause problems for a considerable number of players, as their internet connection struggles to provide the required bandwidth.
Like all developers Bungie is aware of this problem, which is surely one of the reasons why the game is limited to a send rate of 40Hz on PC, as sending updates more frequently means increasing the traffic and so the bandwidth requirements. That was surely also considered when Bungie made the decision that the Crucible should be 4v4 and not 5v5 or 6v6, as more players also mean more traffic.
My verdict
With Destiny, Bungie set out to create a game that blurs the line between singleplayer and multiplayer. And while Bungie clearly accomplished its goal with their "uniquely complicated networking topology," you have to wonder if it was such a good idea to use the same architecture in the Crucible, as peer-to-peer has some very clear downsides that negatively affect your experience as a player.
Back in 2010, when Bungie started to work on Destiny, the server hosting situation was quite different to what we have today, with scalable cloud services like AWS, Azure and Google Cloud. So I wonder if Bungie would use the dedicated server network model if they built Destiny from scratch today. Dedicated servers obviously have many advantages, but they do cost more money to maintain. In that context it will be very interesting to find out which network model EA chooses for Anthem, as that game seems to be quite a bit like Destiny.
The bottom-line is that the PC provides the ultimate Destiny 2 experience as the gameplay is smoother at high frame rates, and the network delay is shorter since the game sends and receives updates at up to 40Hz instead of 30Hz. But it still has its downsides: the game shouts out your IP address to everyone you encounter, overlays from third party tools don’t work, the functionality of gameplay capture tools is in some cases limited, the responsiveness of the hit registration will vary on a player to player basis, and some players will run into issues with the required upstream bandwidth, a problem that rarely happens with dedicated servers.
