What it's like running one of the world's biggest multiplayer games during quarantine

The pace at which Warframe changes and expands with new features is sometimes dizzying. Since October 2019, Warframe has added Shadow of Mordor-style bosses individualized to each player, ship-to-ship combat, a rework of that ship-to-ship combat, a new daily mission series that involves an ongoing murder plot, and a special live event where one squad of players complete objectives on a planet's surface that dynamically change the mission for another group of players fighting in space. Half of those updates happened while most of the world was under quarantine due to the coronavirus.

Big-budget studios have had to delay games to adjust for the complicated new reality of entire development teams now working from home. As head of Xbox Phil Spencer explains, that means shifting deadlines for some of the biggest upcoming games due to the inability to do motion capture, sound recording, or other aspects of development that aren't possible while social distancing.

But for Warframe, one of the PC's biggest and best free-to-play games, it's a lot more complicated than adjusting budgets, internal milestones, and release dates. When Digital Extremes ordered its 320-plus employees to work from home back in March, it still had to operate a live-service game with tens of thousands of players across four platforms logging in each day—all while getting ready to deliver one of Warframe's more ambitious updates. As senior community manager Megan Everett and senior game designer Pablo Alonso tell me, it was uncharted territory, and the pressure was on.

"We were not prepared to have 320 people need to access our database off site," Everett says. "That's just nuts."

Settling in 

March 13, 2020 might feel like it was three years ago, but Everett and Alonso both remember it well: That was the day Digital Extremes management broke the news that everyone would begin the transition to working from home. Ontario, the province in Canada where Digital Extremes is located, had just reported its 79th case of COVID-19. Not even a week later, by the time Digital Extremes had closed the office, the total number of cases had more than doubled. The provincial government declared a state of emergency.

Everett says the news didn't catch anyone by surprise, though. For two weeks rumors had been circulating the studio about what would happen if the epidemic worsened. When plans were finalized and the official order was given, employees had four days to collect what they needed and ease into a very different routine from the one they were familiar with.

But when you're an employee of a large studio like Digital Extremes, working from home isn't as simple as you might think. Developers create assets, like a new gun or character model, which is stored in a network with no access to the outside internet. It's here that all the different parts of Warframe are given shape and slowly come together—programming, art, music, writing, and countless other aspects of its ongoing development. From there, 'builds' are compiled that mash everything together into a playable program that is then uploaded to distribution servers, where it can be downloaded as an update. But everything hinges on developers having reliable access to that central server.

Digital Extremes' HR and IT departments had to work for days on end to get employees' home computers hooked up to the central server through VPNs and greenlit IP addresses. But not everyone has a PC capable of running demanding software like Unreal Engine 4 and 3D sculpting tools like Maya and ZBrush, so many employees also had to take their work PCs home.

In Alonso's case, he left his workstation at the office and uses desktop streaming software to remotely control it from his home PC. It's not ideal.

"It has its pros and cons," he laughs. "It suffers a little bit from video compression, which is not the greatest, and [Warframe] runs at, like, 15 frames per second. I've had a couple of occasions where I'm not super sure if what I'm seeing is an actual issue [with the game] or an issue caused by the way that we're working."

When Alonso is programming an in-game tooltip or sketching down ideas for new warframe abilities, it's not that much of a frustration. But when you're trying to fix microscopic issues with how a warframe feels in the thick of combat, it's impossible. Fortunately, internal builds of Warframe are created several times a day that developers can download and run natively on their home machines if they need to. "So, if I want to test something in a version where I can run at full 4K, 120 frames per second, I need to put in my change, submit it, wait a few hours for a new build to come out, and then I can test that one," Alonso explains.

As you can imagine, that slows things down quite a bit. "You have all these people that are in limbo that can't access what they need," Everett explains. "There was like a little bit of downtime there just to get everyone set up. Everyone was just trying to figure it all out."

Slipping up 

Like a lot of online games, Warframe's developers are constantly talking with the players. Multiple times a week, community managers like Everett play the game on Twitch to an audience of hundreds, and some of the biggest names at the studio, like creative director Steve Sinclair and chief operating officer Sheldon Carter, regularly sit down to talk about what's next during dev streams. Sinclair even livestreamed the final hours of the Fortuna expansion to 40,000 people as the team raced to add the last bit of polish.

Normally, most of these streams happen in Digital Extremes' office with a team of producers so that people like Everett can focus on engaging with viewers. But with the offices closed and the release of the next update coming closer, Everett needed to find a way to do everything from home by herself.

After retrieving as much equipment as she could from the office, she set up her own livestreaming station in her second bedroom. "I knew absolutely nothing," Everett laughs. "So once I was set up at home, I kind of had to learn all that, and it was very stressful and very sweaty and scary."

While streaming might seem simple to a lot of people, Warframe's weekly dev streams fall into the higher end of the production spectrum, with scene transitions, complicated capture setups, and an enormous audience with high expectations. "Once I got that all figured out, I was going to do a Monday stream," Everett explains. "I took some time on Sunday evening and was like okay, I'm going to set up my OBS [app for streaming] and make sure I have everything because I was so scared of messing it up."

That's exactly what she did. While tinkering with OBS, Everett accidentally turned on the stream. Suddenly, thousands of Twitch users got a notification that the official Warframe channel was live, but when they tuned in, all they saw was a static image saying the stream would be starting soon and the sound of an Animal Crossing YouTube video in the background. "So it was like a 'starting soon' screen with an Animal Crossing video in the background for like six minutes and [people were] like, what is this? I was mortified that I had done that because I had no idea what it was doing. I couldn't sleep that night."

A video of the accidental stream today (Isabelle Prime confirmed) from r/Warframe

Though Alonso hasn't accidentally broadcast Animal Crossing walkthroughs to thousands of players, he's had his own struggles being physically isolated from his coworkers. He says a lot of the Digital Extremes development team is accustomed to using programs like Slack to communicate, but there are moments when text chat just won't cut it.

Protea is one of the newest warframes coming in the next big update, and her ultimate ability involves manipulating time. Alonso says figuring out how that ability should work was an enormous challenge because, well, time manipulation is complicated. Heaps of information, like the player's position, velocity, rotation, and any interactions they have with the environment has to be stored in a big list so that when the ability rewinds itself, all of those actions can play back in reverse order. With other players still moving forward in time around you, there's "a lot of moving parts," Alonso says.

"Normally what I would have done is ask the main ability programmer to come over to my office and we would have chatted it out and made a couple quick drawings in the notepad. The end, right?" Alonso explains. "It definitely gets hard to talk about more complex things just over chat, though. I literally ended up making like a drawing in Photoshop with like, little diagrams with dots, numbers, the distances, and some math. That's how we had to figure it out—it definitely was slower than it would have been. Even now that we did the diagram, I'm not fully convinced that we're on the same page."

Issues like this might not seem that big, but across a company of 320-plus people, it can add up. Though both Alonso and Everett say that things have been getting easier over time, they also admit it played a part in why some of Warframe's biggest updates of the spring were delayed, and why one update in particular was a bit of a disaster.

Buckling down

Operation: Scarlet Spear was meant to complete the trifecta of features showcased during Empyrean's initial reveal over a year ago. Constantly changing space villains and ship-to-ship combat were the first two parts, with Scarlet Spear adding Squad Link, a feature where a group of players could fight on the ground and tackle objectives that, when completed, would change the battlefield for a different squad fighting in space. When it was revealed last year, the example mission demonstrated this feature by having the ground squad disable the energy shield of an enemy capital ship so players in space could board it. Alonso says the update introduced some under-the-hood tech that Digital Extremes hadn't used before, like a new central player hub that acted as a staging point for all these squads heading out on ground and space missions.

When Scarlet Spear released on March 24, it was a perfect storm of problems: In addition to the chaos of developers adjusting to working from home and new technology supporting the update, Digital Extremes wasn't able to rigorously test how certain features would handle the stress of thousands of players jumping in all at once.

"Scarlet Spear was an absolute mess," Alonso says. "That update was maybe overly ambitious and probably would have been also super hard even if the whole move to home wouldn't have happened. But then you pile that on and it just got really out of control."

Warframe is a game that continually takes risks that don't always pay off, but Scarlet Spear was especially broken. Matchmaking issues made it nearly impossible for some players to embark on the new mission types, bonus rewards often didn't trigger because the scoring system often broke alongside some objectives, and the grind for rewards was way too heavy. Making everything worse, Scarlet Spear was a limited time event that only lasted until April 28. "We weren't able to test every single scenario," Everett admits. "So when we put out this Operation, and you get thousands of people playing it, they're obviously going to find things that we weren't able to find, and a lot of it was pretty bad in terms of matchmaking and stuff."

Though things did improve over time as Digital Extremes made more fixes, many players still weren't happy about the state of Scarlet Spear. Everett says the team has already committed to running the event again in the future to make up for it, and the studio has expanded its public test server to get more player feedback earlier in the development process. "Operation Scarlet Spear was not perfect, and it was a rough ride," Everett says. "But I'm still proud that we've put it out during the whole situation that we're in. And I think people really understand that."

It's one of the benefits of Digital Extremes' transparent relationship with their players. Days after Scarlet Spear launched, Sinclair hopped on stream with live ops and community director Rebecca Ford to talk frankly about what was going wrong and why, and how Digital Extremes was going to fix it.

But that was over two months ago, and even if Scarlet Spear goes down as one of Warframe's darker moments, Digital Extremes hasn't had much time to sulk. Since the event ended, Warframe has added the new Night Wave series—a clever take on daily missions—where players solve a murder mystery. This week comes The Deadlock Protocol, which reworks an entire enemy faction, adds a new warframe to collect, and new chunks of lore and story. Though release dates have been pushed back slightly with most of these updates, quarantine has only slowed Digital Extremes down by just a few weeks.

At the same time, Everett says the team is getting increasingly proficient working at home, but the social isolation is still a challenge. "You don't realize how much physical and social contact plays into your daily routine until it's literally gone," she says. "Even if I went to work and said nothing to anybody all day, I would at least still be [comforted] by their physical presence and hearing their voices and laughing, and that would at least do something for me."

"Do I actually miss seeing my coworkers? Absolutely." Alonso says. "Just to have lunch with [some of my coworkers] pretty much every day, and now we just send each other pictures of what we're eating."

Both Alonso and Everett say it's those little gestures—text messages to check in on a coworker or random video calls just to say hello—that have helped fill the void of no longer working in an office. It's a weird, often uncomfortable situation, but it's also helped both realize what truly matters.

"Even though it is kind of terrifying where we're at with all the stuff that's happening, At the same time, I've never shared something like this with so many people," Alonso says. "Like, literally every country in the world is dealing with the same thing right now right now. There's something kind of—I don't want to say beautiful because obviously it's a horrible situation—but at the same time there is something so special about it that is like, we're sharing a problem with billions of people, and we're all trying to kind of find our way through it."