I'm getting a little tired of our deep-learning future, folks. While generative AI and deep learning technology isn't inherently bad—it's being used for folding proteins and advancing medical science, for example—the routine doling-out of slop has everyone paranoid, content trawlers are scraping through the web sucking up everything in sight, and an endless parade of techbros are insisting their software can think, feel, and make important decisions, despite evidence to the contrary.
An anonymous coder, writing to 404media under the fake name Aaron B, has also been feeling the malaise, and he's doing something about it. Aaron B has created a digital flytrap to keep web-crawling bots in a thus-far undetectable doom loop, scanning the same pages over and over until someone human comes over and fixes it.
The program, dubbed Nepenthes (after the tropical pitcher plant), exploits a weakness in these crawlers—the fact that they're apparently really dumb, as Aaron B explains:
"It's less like flypaper and more an infinite maze holding a minotaur, except the crawler is the minotaur that cannot get out. The typical web crawler doesn't appear to have a lot of logic. It downloads a URL, and if it sees links to other URLs, it downloads those too. Nepenthes generates random links that always point back to itself—the crawler downloads those new links. Nepenthes happily just returns more and more lists of links pointing back to itself."
The site itself lays it out in full, as well as the usage cases—allowing you act defensively to "flood out valid URLs within your site's domain name, making it unlikely the crawler will access real content." You can also, as the site suggests, play offense by ignoring the code's existing list of known trawler IPs and allowing bots to "suck down as much bullsh*t as they have diskspace for, and choke on it."
There's also a link to a demo of the software in action, as well as an all-caps warning that "THIS IS DELIBERATELY MALICIOUS SOFTWARE INTENDED TO CAUSE HARMFUL ACTIVITY" and that you will experience "significant CPU load" if you use it. Govern yourselves accordingly.
This isn't a perfect solution to the plague of AI scrapers—most AI models guzzle a huge amount of fuel, and Nepenthes doesn't do anything to stop that: "They are still consuming resources, spinning around doing nothing helpful, unless they find a way to detect that they are stuck in this loop."
Essentially, there's sort of a dark feedback loop that's possible here—these scraping bot farms we're all very tired of could be motivated to add more scrapers to counteract them, driving up energy costs and… you know, accelerating the horrors. It's very much a 'fine, we'll both burn' stopgap. Aaron B even couches it in such terms:
"It's also sort of an art work, just me unleashing sheer unadulterated rage at how things are going. I was just sick and tired of how the internet is evolving into a money extraction panopticon, how the world as a whole is slipping into fascism and oligarchs are calling all the shots—and it's gotten bad enough we can't boycott or vote our way out, we have to start causing real pain to those above for any change to occur."
As for the claims that bots can skip over these traps, Aaron B seems confident he's at least wasting some of their time: "I've several million lines of access log that says even Google Almighty didn’t graduate."
