This week's internet blackout was caused by a single user updating their settings
A bug, dormant since mid-May, was triggered by a very specific user configuration update.
Fastly, one of the internet's largest content delivery networks (CDN), went down this Tuesday, resulting in millions of users being unable to access certain websites. 85% of the network covered by Fastly returned errors on Tuesday morning, and it was all caused by a single user configuration update that uncovered a bug that had laid dormant in Fastly's infrastructure since mid-May.
In an interview with The Guardian, the head of infrastructure and engineering at Fastly, Nick Rockwell, explained what had actually happened to bring down its services and also apologized for the disruption. It feels rare for such a big company to be so transparent about this, but it's certainly something to welcome.
Content delivery networks operate on the principle that the internet is faster and more stable if the users are physically closer to them. This results in faster downloads, better security, and a host of other features.
It also means there's a point of failure if something goes wrong though, and that's exactly what happened on June 8.
Best chair for gaming: the top gaming chairs around
Best gaming desk: the ultimate PC podiums
Best PC controller: sit back, relax, and get your game on
“On May 12, we began a software deployment that introduced a bug that could be triggered by a specific customer configuration under specific circumstances,” Rockwell told the Guardian. “Early June 8, a customer pushed a valid configuration change that included the specific circumstances that triggered the bug, which caused 85% of our network to return errors."
Essentially a bug had been introduced into the system on May 12 but had laid dormant until a customer updated their settings on June 8, which triggered the flaw, taking down most of the internet with it—including PC Gamer—for many users. Fastly spotted the problem within a minute, and "within 49 minutes, 95% of our network was operating as normal.”
There's a financial impact to users not being able to access sites of course, and SEO agency Reboot estimates that the downtime cost Amazon $32M in sales. Gulp.
The biggest gaming news, reviews and hardware deals
Keep up to date with the most important stories and the best deals, as picked by the PC Gamer team.
Alan has been writing about PC tech since before 3D graphics cards existed, and still vividly recalls having to fight with MS-DOS just to get games to load. He fondly remembers the killer combo of a Matrox Millenium and 3dfx Voodoo, and seeing Lara Croft in 3D for the first time. He's very glad hardware has advanced as much as it has though, and is particularly happy when putting the latest M.2 NVMe SSDs, AMD processors, and laptops through their paces. He has a long-lasting Magic: The Gathering obsession but limits this to MTG Arena these days.