Good news for Destiny 2 players who've been struggling with Beavers recently: Bungie has finally brought the gnawing little beasties to heel, with some able help from Valve.
The Beavers in question, I should probably clarify, are Destiny 2 error codes. Rather than opting for mundane terminology that might hint at what the problem is, Bungie uses nicknames like Buffalo, Saxophone, Chive, Zebra, and Beaver, which has recently been rearing its toothy little head for some players. Beaver errors, along with Flatworm and Leopard, "are caused by a failure to connect your console to another player’s console via the internet," the Bungie help page says.
But figuring out what exactly was causing the spike in Beaver errors has been a real chore, as Valve software developer Fletcher Dunn explained, in great detail, on Twitter. He got involved in the process because Destiny 2 has been making use of peer-to-peer networking technology on Steam that he's been working on.
The tech is software-based routing: route through general-compute linux boxes using custom protocol, with clients exercising significant routing autonomy. Since peers do not share their IP addresses, script kiddies cannot DDoS other players.https://t.co/Uxp8hgThZXJuly 23, 2020
That resistance to DDoS attacks that Dunn describes in the tweet above sounds like an obvious big plus, but Dunn added that the disconnection rate—those pesky Beavers—was higher than expected after it launched.
"Some players would get it often; others never saw it. Restarting the game would often make it go away," he wrote. "We could never reproduce it. (It works on my machine!)"
After a couple of months of fixing bugs, working with Bungie, and bonding with his sons over a lot of Destiny 2, this happened:
The kernel does a heroic amount of work for you to deliver a single UDP packet, most of which is bypassed by XDP. So XDP is *insanely* fast. In our case, the XDP code can process 5x-10x the packets for the same CPU cost as the plain BSD socket code.July 23, 2020
It all gets quite technical, but the short version is that the discovery of an "oddball" relay unfolded into the realization that a bug was causing a communications breakdown under very specific circumstances, which in turn triggered another bug that kept any of it from being logged.
Bungie confirmed that the fix is in place and works, although you may still see Beavers popping up now and then.
"This past week Valve identified hardware configuration issues with 4 relays in their Chicago, Virginia, Stockholm, and Dubai data centers. In each case, the affected relay was unable to send traffic to one other relay in the same data center. If a connection to a peer went through both of those relays, then it would drop. Valve has fixed the configuration issues, and we have confirmed that the rate of disconnections in the affected areas has been reduced significantly," it wrote in the latest This Week at Bungie update.
"These changes have reduced Beaver error codes significantly with players in the central and eastern United States of America being impacted the most. Beaver error codes are an indication of connectivity failure and can be caused by a wide range of issues. Some Beaver error codes are caused by local internet connectivity issues. If you still experiencing a lot of Beaver errors please use follow our Network Troubleshooting Guide and then report any further issues on our Help forum."
As for why it took months to figure all of this out, Dunn summed it up nicely:
Why didn’t monitoring catch this?We do have significant monitoring, especially for problems *between* data centers. But it did not detect problems between hosts in the same data center.Also: networking is complicated.🧵>July 23, 2020
May the Mother Beaver watch over us all.
