Why EVE Online's 'million dollar battle' was a huge bust

Update: Initially, this story said that EVE's server was randomly prioritizing calls made to the server from player's clients. CCP community manager Paul Elsy has since clarified that was not the case and the issue is slightly more complex.

"To clarify, the EVE Client will make a call to Tranquility and the server will put it into its queue of tasks. TQ might come back to the request more than once, for instance, if it needs to call the database for information while processing, then it will suspend the task and come back to it again once the database call completes and it has the information.

"When it is finally able to complete that task, then it will reply to the client eventually, but if the client has been waiting for too long then it may have disconnected (possibly because of the load on the server) and there is no one to listen to the reply from TQ.

"From your perspective as a player, the action was not completed as far as you know, or it may appear that stuff’s happening randomly, but the server will have completed what it could, it just couldn’t relay that to your client given that you’re disconnected due to heavy load."

We've changed original statement to reflect this clarification.

Original Story: Earlier this week, over 6,100 pilots piled into the star system of 9-4RP2 to engage in the biggest battle in EVE Online's history. Outside of 9-4, another 6,000 pilots brawled for control of supply lines or dutifully waited for the call to reinforce their allies. Over ten thousand people were watching on Twitch, and even the mainstream media took notice. It was supposed to be the most destructive battle ever seen, dwarfing the losses incurred by the previous record-holder, the Bloodbath of B-R5RB. Players wanted a slaughter, but only a few hours later it was over. The total damages? About $10,000 USD. EVE's "million dollar battle" fell a little short. 

Great expectations 

As I already wrote, the Battle of 9-4RP2 was fought over a single Keepstar Citadel. This astronomically expensive space station is owned by Pandemic Horde, an alliance belonging to a group loosely referred to as The North. Traditionally, however, The North isn't Pandemic Horde's home. Back in 2016, a massive coalition (including Pandemic Horde) were rallied under the banner of the Moneybadger Coalition to unseat The Imperium, once considered EVE's most unassailable empire. After a crippling betrayal, The Imperium forfeited their northern empire and fled to the south-western corner of New Eden, EVE's virtual universe. It was a devastating defeat.

For over a year The Imperium and The North have been engaged in an arms race, stockpiling powerful Titan supercapital ships valued at over 100 billion ISK (around $1,300 USD when converted to EVE subscription time). When The Imperium began mobilizing its armies and heading toward their own homeland, players got excited: Two of the biggest factions in the game, each packing unimaginably large fleets of Titans and even bigger fleets of sub-capital ships, were about to meet. 

Over the course of weeks, The Imperium launched several successful skirmish fleets and managed to weaken the defenses of Pandemic Horde's 9-4RP2 Keepstar. This past Tuesday was their chance to deal the killing blow. In response, the entirety of The North rallied to aid Pandemic Horde and prevent The Imperium from securing a vital beachhead.

When they met on the field of battle two days ago, players weren't just expecting to see each side deploy the full might of their supercapital fleets, they wanted it. Being a sandbox MMO, players in EVE Online have to make their own fun. It's a burden carried by the leaders of these thousand-player alliances to continually keep their pilots entertained with new wars, fights, and reasons to play each day. And nothing gets EVE players (and the wider gaming community) more excited than a massive, galaxy-shaping battle. They could have gotten just that, if only EVE Online's servers could manage the overwhelming stress of 6,100 players shooting each other.

"It's always a learning experience," CCP Games community manager Paul 'CCP Falcon' Elsy tells me. He says that whenever EVE Online has a massive battle like this, it's all hands on deck for the developers.

EVE Online differs from your typical MMO in that it doesn't spread its playerbase across multiple servers like World of Warcraft. Instead, everyone shares one massive world and, from a server architecture standpoint, that creates some unique challenges. EVE Online is powered by a legitimate supercomputer boasting several Intel E5-2637v3 cores and 256GB of RAM. Elsy tells me that during the battle, they saw the CPU cores associated with that particular area of digital space reach 100 percent load, using almost 64GB of RAM to process each one of those 6,100 players' actions.

But managing that kind of load in real-time is impossible. That's why EVE Online uses Time Dilation ('TiDi') to slow things down during big conflicts to give the servers a chance to process all the action without going belly up. Under the full effect of TiDi, one second is actually ten seconds. But, as pilots from both sides of the conflict tell me, the result was less than ideal.

Fog of war 

Sort Dragon was one of the supporting defending generals on the field that day. He's the CEO of Darkness and leader of the Guardians of the Galaxy, one of two coalitions that make up The North. As he tells me, both sides were almost completely crippled by the sheer amount of server lag. "Players were useless and could not do anything at all except stare at a login screen or stare at their ships unable to do anything," he says.

Not only were CCP's servers being hammered, but client-side, players' computers struggled to render 6,100 individual ships all bundled up on screen. Early into the fight, players started complaining that the game was crashing or they were stuck in loading screens. Another pilot flying for Pandemic Legion, JakeMeister, tells me that ships were unresponsive and could take minutes to process a single command like firing a gun. Twitch streamers broadcasting the fight couldn't even move their camera without risking a crash. With the added effect of TiDi slowing things down, the action on screen played out at a crawl making it impossible to tell if the game was even responding. It was a nightmare. 

"As a leader, I felt helpless," Sort Dragon tells me. "It was commented on today that I didn't talk much on comms [during the fight] and the honest fact was I didn't because I couldn't give people much hope."

Speaking to Paul Elsy, he confirmed that EVE Online's servers weren't functioning as they should. Each time a player takes an action, like firing a cannon, a call is sent to the server to be processed so that action can be carried out. Under load, these calls should ideally form a queue and the server will work to complete them in chronological order—albeit very slowly. But the heavy load meant some calls weren't completed in a coherent way, leaving some players feeling like their actions were being ignored. It was impossible for commanders to coordinate their fleets effectively. It's something that CCP says they're investigating.

The likely culprit at this time is changes that were made to carrier and supercarrier ships. These behemoths field waves of fighter drones that the player controls, and recent changes were made to group these drones in 'flights' to help mitigate the processing power required to control them. But players found that they were malfunctioning and they often couldn't load a new set of drones in their launch tubes to replace destroyed ones. For the defenders, it was a major irritation, but for The Imperium, it was a disaster. Their entire strategy hinged on using these drones.

Earlier in the week, The Imperium installed a smaller Fortizar Citadel on the doorstep of Pandemic Horde's Keepstar. The backbone of their strategy was to keep their carriers and supercarriers protected within range of the Fortizar and use long-range bomber drones to deal damage to the Keepstar. But once the fighter drones stopped functioning correctly, the added stress of server lag and TiDi made it impossible to pivot and adapt to a new plan. "Our plan was to ferry out replacement [fighters] to our ships and replace the ones that were destroyed," Asher Elias, The Imperium's head fleet commander tells me. "Unfortunately the server broke in such a way that not only could we not replace our fighters, we couldn't even launch the second wave of fighters we had in our carriers. So we couldn't even use the full capacity inside our ships much less replace them."

"Unfortunately the game mechanics didn't work as intended and they didn't end up getting what they wanted to out of the fight," Elsy says.

That wasn't the only hurdle The Imperium had to tackle, either. To encourage fair fights and give time to prepare, Citadels have pre-determined vulnerability windows. The real battle started the moment Pandemic Horde's Keepstar entered its final 15-minute timer. It's complex system, but the gist of it is that as long as the attackers maintain a steady stream of damage to the Citadel, its timer will remain paused until it's destroyed. The defenders must repel the attackers to lower the damage done to the station long enough for the 15 minutes to lapse, at which point the Citadel automatically heals all damage dealt and becomes invulnerable for a time.

Here's the kicker: The timer isn't affected by the slowdown of TiDi. That means The Imperium had to deal effectively ten times as much damage as normal, since each of their actions was slowed by an order of ten. I asked Elsy if CCP felt the system was working as intended. "It's by design due to a few technical restrictions that we have, but we've been chatting a lot with our game design team just to roll over what we've learned from the fight and what we can change going forward. I can't say we're looking to address it immediately, but it is a point of discussion because it does give the attackers a significant disadvantage."

Doomsday delayed 

Before the fight began, the expectation was that both sides would escalate the conflict into a complete slaughterhouse. In EVE, all-powerful Titans are often played like a trump card. Each side commits more and more expensive fleets to the field, raising each others bets until they have no option but to either retreat or drop their Titans into the mess. That's when things will get expensive. With both sides reportedly having hundreds Titans ready to go, players were hoping the damages would reach a staggering $1 million USD.

But from The Imperium's perspective, there was never any reason to deploy Titans. With pilots struggling to even log in—not to mention fire their guns—there was no guarantee that deploying their Titan fleets would be effective. What's more, the northern defenders cast their bet early and deployed some of their Titans to the field. If The Imperium responded by unleashing theirs, there was a significant chance they would all be destroyed before their pilots could even load into the game. That kind of loss would be enough to end the war before it ever began.

"It was frustrating to a degree," Asher Elias says. "I've played the game a long time and I've been in plenty of giant fights. Some of them are slow but everything works and that gives you the kind of time to make tactical calls you couldn't in other games while others are slow and frustrating like this particular fight. There was a dawning realization that we were fighting the game itself more than our enemy and there was very little we could do to win."

Sort Dragon doesn't see it that way, though. "The fight was [The Imperium's] to lose," he says. Between the lag and technical difficulties, The North thought they wouldn't be able to mount a capable defense. "We couldn't believe we survived and we couldn't believe they didn't commit. When [The Imperium] went into the fight they knew it was going to be hard but they knew they wanted the challenge. Do I believe they got the short straw? Maybe. But I do know they went in knowing what to expect."

Asher Elias, on the other hand, doesn't think it's that simple. "Both sides had the opportunity to fight their enemy Titan fleet on their own respective Citadels, it's a pretty rare chance and both sides elected to pass on the opportunity to focus on attacking and defending the objective. EVE is all about the meta-game. If Sort Dragon thinks he can goad us into a foolhardy attack next time by calling us out in PC Gamer, it would be a big win for him. Don't expect us to make that kind of mistake."

After a few hours, it became clear that The Imperium was unable to do enough damage to keep the timer paused. Eventually the call went out across fleets to get out if you could and regroup. It was an anti-climatic ending to a battle tens of thousands of viewers and participants were hoping would spark the next great EVE war. And, understandably, many point to CCP Games as the culprit. "The thing is, if we prepare for 4,000 pilots, each side will bring 5,000," Elsy contests, adding that CCP has a unique challenge in trying to make fights like these workable. "There's just nothing like this happening in other games."

It's easy to sympathize with the technical challenges facing CCP during these moments, but the result is still disappointing. It's not often EVE Online makes headlines in mainstream media, and having that much attention drawn to a single battle only to have it deflate into nothing has the community pent up with unspent bloodlust. It's been four years since The Bloodbath of B-R5RB where 11 trillion ISK (an estimated $300,000 USD) went up in smoke. While the number of participants in conflicts has risen dramatically, players haven't seen that kind of destruction since. Veteran pilots like Asher Elias point the finger at CCP's game design, which he argues no longer favors these kinds of epic clashes. "The mechanics are a bit arcane here, but because of the way Citadels work the side [defending] a Citadel has an immense advantage," Asher says. 

The conflict isn't over, though. While the Keepstar was saved, The Imperium can attack it again in about a week's time. If it's serious about retaking the North from its enemies, another major fight is inevitable. Whether 6,100 pilots show up and Titans are dropped, finally giving everyone the fight they've been wanted for years, no one can say. Elsy is hopeful, though. "I was talking with a few guys from The Imperium yesterday before the fight kicked off," he says. "You don't march that amount of force north with the intention of going home if the first battle goes sour."

We'll have more on this conflict as it evolves over the coming weeks.