How Rust got back online after its servers literally caught fire

A view of firefighters battling a blaze at the OVH data center in Strasbourg, France on March 10th, 2021
(Image credit: Sapeurs-pompiers du Bas-Rhin)

When Rust producer Alistair McFarlane went to bed well past midnight, it seemed like a brief outage of Rust's European servers was annoying and abnormal, but not severe. The Facepunch team in the US would be able to periodically check on the problem and bring the servers back up as soon as they could. "These problems never last more than a few hours," McFarlane told PC Gamer in an email.

Rust is one of the biggest PvP survival games, a mainstay of the genre and of Steam's most-played games list for years now. Each of the 24 servers that had gone down held nearly 10,000 people during peak hours, so more than a few players were inconvenienced.

By the time McFarlane was awake at 7 am, his phone was flooded with messages about what had occurred overnight: A fire at an OVH datacenter had affected their servers. It couldn't be too bad, he figured. "Data centres these days are built to high standards with advanced fire suppression systems," said McFarlane. 

When he got to his PC, however, he discovered that it was far worse than he'd imagined. A massive fire had gutted the Strasbourg data center hosting Rust's EU servers

It was a fire so large that a pump boat was deployed to ensure enough water could flow to the local firefighters. They'd been fighting it through the night and were still pumping water onto the site the next morning. Nobody was injured, but potentially millions of websites were offline. 

(Image credit: Sapeurs-pompiers du Bas-Rhin)

"I was utterly shocked when I saw the pictures on Twitter of the building fully engulfed on fire," said McFarlane. "It was clear we needed a plan to get the affected servers back online and communicate to our players."

Simply identifying which servers were affected wasn't easy. "I'd guess everyone was trying to log in at once to find any information," he said, which meant that the server company's web panel nearly unresponsive. 

"We confirmed a total loss of our servers and immediately started sourcing new servers," said McFarlane. That was less simple than they'd have liked. "We were not the only ones doing this, stock of OVH servers in other locations began selling out," he continued. Complicating things, Rust's servers see large DDoS attacks regularly, and as a result they're particular about the hardware they use. On top of the sudden scarcity, they couldn't take just any servers.

"We had to settle for OVH Poland," he said. That's some 1,000 kilometers from Strasbourg, the server's original location.  It wasn't ideal for Rust's players, or for McFarlane's team. "We'll need to move the servers again in the coming weeks, but it's all we could source with the quantity we required quickly," he said.

A view of firefighters battling a blaze at the OVH data center in Strasbourg, France on March 10th, 2021

(Image credit: Sapeurs-pompiers du Bas-Rhin)

Within 11 hours of the first Rust servers going down, the Facepunch team had gotten all of them back up and running. That was just four hours after they became fully aware of the fire. "We're always prepared to have servers go down but never en masse," said McFarlane.

As for the data on the servers, Facepunch had both local and on-site backups, as well as offsite backups, but player progression data wasn't backed up offsite. "Lessons learnt," said McFarlane. "We need to ensure player progression data is backed up offsite, and we'll make sure this happens going forward."

"I feel we did everything else right," he continued. "We did the best we could with the time we had, running on the little sleep we had." 

Some players were upset that they lost days of progress in a game that runs in real time whether you're logged in or not, but McFarlane said that, in general, players understood that it was an exceptional situation.

McFarlane estimated that around 30 dedicated Rust servers were lost during the fire, though all of those are now back online. It's unknown how the fire started yet, or whether OVH was at fault. Regardless, McFarlane says OVH has done well for Facepunch in the past and that it'll continue using OVH in the future.

Contributor

Jon Bolding is a games writer and critic with an extensive background in strategy games. When he's not on his PC, he can be found playing every tabletop game under the sun.

Read more
Palworld Pal hammering on rocks with a pickaxe
'He was trying his best'—Palworld had a lone server guy trying to keep the game afloat during its 2 million player launch
A goblin with sharp teeth, wearing goggles, lets out a mischievous cackle in WoW's latest patch: Undermine(d).
The hooligan hacker guild that tore up WoW's newest raid (twice) just posted video evidence of the whole thing, and it's got me feeling weirdly nostalgic
Path of Exile 2 early access class key art
Around 66 accounts in Path of Exile 2 were compromised, due to a one-two punch of an old unused Steam account and a backend bug
Crashing servers, flame wars, and a 60-day path to redemption—the utterly chaotic first year of Helldivers 2 has been a democratic doozy
A hand holding a compass in the woods
Players are already finding creative solutions to Prologue: Go Wayback's grueling difficulty: 'You could just chug gasoline'
A view over a peaceful looking Minecraft farm, boasting fields of crops.
It's been a big year for factory sims, but instead of getting a degree in engineering, I've decided to go back to where it all started for me: modded Minecraft
Latest in Adventure
Inside
Limbo and Inside studio demands compensation from co-founder Dino Patti for alleged 'unauthorized use of Playdead's trademarks and copyrighted works'
Two characters sitting on a bench talking
Wanderstop review
Zoe showing off in front of Mio
Split Fiction review
Rusty Rabbit chomping a carrot like a cigar
Rusty Rabbit turns Yakuza's Kazuma Kiryu into a fluffy bunny
Pathologic 3 screenshot
Get ready to get weird in Pathologic 3: Quarantine, a free 'prologue chapter' about a young doctor looking for immortality in the world's most miserable town
A young woman's face bathed in light
Lost Records: Bloom & Rage Tape 1 review
Latest in News
Mechs fight on the outside of a spaceship
MechWarrior 5: Clans is getting DLC with playable Elementals and a fight on the outside of a spaceship
Aloy - Horizon
'I feel worried about this art form:' Unsurprisingly, the real Aloy from Horizon isn't a fan of AI Aloy
Crying laughing emoji with disturbing realistic elements for REPO
REPO's first update will add a new map and a 'duck bucket' so we can finally give that pesky quacker a time out
Man facing camera
The Day Before studio reportedly sues Russian website for calling infamous disaster-game a 'scam'
Will Poulter holding a CD ROM
'What are most games about? Killing': Black Mirror Season 7 includes a follow-up to 2018 interactive film Bandersnatch
Casper Van Dien in Starship Troopers
Sony, which is making a Helldivers 2 movie, is also making a new Starship Troopers movie, but it's not based on the Starship Troopers movie we already have