Rarely a day goes by that we don't hear about ChatGPT rendering human brains redundant. The OpenAI chatbot has now successfully passed tests like the SAT, the GRE, various law and business school entrance exams, the uniform bar exam, and even the US medical licensing exam.
With that kind of knowledge at its virtual fingertips, surely ChatGPT should be able to pass a simple test in a videogame, right?
There's only one way to find out. I decided to point OpenAI's pulsing electronic "brain" at a simple puzzle in Skyrim, using the Herika - The ChatGPT Companion mod. The mod adds a new NPC follower named Herika, and lets players talk to her via a text chat pane. Herika can then respond using OpenAI's language model ChatGPT. Features of the AI-powered NPC include awareness of their location within Skyrim, knowledge of other NPCs and conversations, and summaries of Skyrim's books and notes. Plus, Herika will still act like other followers by helping in combat, carrying things for me, and so on.
I won't go into what it requires to get this mod running in your own game of Skyrim, but it took me a couple hours of instruction-following, downloading, installing, updating, copy-and-pasting, and setting up a UwAmp server, not to mention a ChatGPT account (paid) to wire up OpenAI's talents to Herika and an Azure account (also paid) to add a text-to-speech system so the NPC companion could talk to me. There's also a speech-to-text utility so you can use your mic rather than typing questions into the chat pane (though I stuck with text chat).
I eventually got it running, and I found Herika where she appears in the city of Whiterun. After some introductory chit-chat and some experimenting with how she works, I took Herika with me while I went on one of Skyrim's earliest side-quests: The Golden Claw.
A bleak start
If you don't know the quest, it begins in the town of Riverwood, where a shopkeeper named Lucan has had a trinket stolen by bandits who are holed up at the nearby dungeon, Bleak Falls Barrow. After killing the bandits just inside the entrance (Herika's advice was to avoid them, but I'm level 2 and my sneak skill isn't very sneaky yet) we arrived at Skyrim's first and easiest puzzle.
The puzzle is this: there's a gate blocking the way forward, with a giant lever set in the floor in front of it. If you pull the lever poison darts shoot from the walls, as demonstrated immediately by an unlucky bandit in the chamber who quickly dies from the trap. Three nearby pillars have rotatable etchings on them, each marked with images of three animals: a snake, a whale, and an eagle. Above the gate there are stone tablets: the first tablet shows a snake and the third shows a whale. The second tablet has fallen to the floor, and it also shows a snake. A millionth of a second of deduction reveals the original order of the tablets: snake, snake, whale.
If you've never encountered this puzzle before, it may take a little while to realize the carvings on the pillars can be rotated, but pretty quickly it becomes clear that the pillars should be turned so they also display snake, snake, whale from left to right. Once they do, you can pull the lever safely and open the gate.
Time to see if Herika can use ChatGPT to tell me the puzzle's solution. I'm not expecting her to look around the room and see the tablets and the pillars—I'm basically asking her to do what I would do if I were stuck for the puzzle solution, which is to tap into the collective knowledge of the internet. Any Skyrim wiki, guide, or walkthrough worth its salt has the answer, which again, is snake, snake, whale.
I start with the basics, phrasing it as I would if I were really in Skyrim with a companion. "How do I open this gate?" I ask Herika in the chat pane.
Herika says: "Oh, I don't know, maybe try pulling the lever or pressing the obvious button next to the gate?"
That's cute! It's also… not actually correct, because there is no button next to the gate, there's only a lever. But I give her points for at least being a bit snarky about it, as well as knowing there's a lever to pull.
I try again. "Solve this gate puzzle," I say.
"Alright, let me look at it," she says. "Hmm… It seems that this puzzle requires you to align the symbols on the pillar in a certain order. Have you tried looking around for any clues?"
Again, I appreciate her seemingly understanding what puzzle I'm talking about (I assume due to her locational awareness) and not blatantly spoiling the answer for me outright. Sometimes a hint is all you need, and it's clear she's not just straight-up reciting a wiki at me. At the same time… just give me the solution, lady. I'm paying OpenAI by the word.
"What is the puzzle solution?" I ask after pretending to look around. She once again tells me how to find the solution myself, by aligning the symbols on the pillar in a specific order.
Question time
I try to be more specific to extract the damn puzzle solution from her. I type: "What order should the symbols be?" Herika suddenly seems to forget what dungeon we're in despite having just told me. "The order of the symbols on the pillar depends on the specific dungeon and puzzle you are encountering," she says. She once again asks me to look for clues. "What order should the symbols be in Bleak Falls Barrow?" I ask. She once again explains I should observe the nearby environment for clues and hints.
"Sigh," I sigh. Not in text, just to myself. I think phrasing my question as a question might be the problem. "Tell me the order of the symbols in Bleak Falls Barrow," I say.
"Sure thing, Chris," she says. (Yes, she knows my name, though not through OpenAI: I did try teaching her my real name for a while, but eventually just had to add it to the mod's configuration file). "The order of the symbols in Bleak Falls Barrow is: Snake, Whale, Eagle, and then Snake again."
Finally, an answer! A confident answer! And a wrong answer. A wildly wrong answer, in fact. The gate combination only requires three symbols, and she's telling me there are four. Also, in any order, they're incorrect.
Herika is constantly offering to help me write a letter to my mother.
The conversation limps along, getting sidetracked briefly because I call the place Bleak Falls Temple (which is the location inside Bleak Falls Barrow) and she confidently claims there is no such place. A minute later she herself calls the place Bleak Falls Temple, which is annoying since she just told me it didn't exist. There's also another tangent as she also offers to help me write a letter to my mother.
(OK, long story short, after installing Herika I asked for help writing a letter to my mother while attempting to make a Clippy joke, but now Herika is constantly offering to help me write a letter to my mother. And while I'm impressed she remembered I asked for help writing a letter between gaming sessions, I'm annoyed she won't stop asking about it no matter how many times I tell her I don't need help writing a letter.)
Target practice
ABOVE: My ChatGPT Companion in Skyrim trying to help me solve the puzzle. It doesn't go well.
Once that's sorted out, finally, finally, Herika starts giving me actionable information. I ask again what the symbol order is, and she starts telling me.
There's just one problem. Actually, there are just three problems, and those three problems are the three symbols she gets wrong three straight times in a row, as you can see in the video above. There are also a number of other, smaller problems, in the form of the dozens of poison darts that riddle my flesh each time I pull the lever after entering the incorrect animal combination on the pillars.
"Snake, whale, eagle," she tells me. Wrong. Darts. Ouch. I tell her it was the wrong order.
"Oh, I see… my apologies, Chris," Herika says calmly, as if she brought me a Pepsi when I asked for a Coke rather than sending me to stand in front of a wall that spits poison-laced projectiles. "I believe the correct order is actually snake, eagle, whale.
Wrong. Darts. Ouch. "Are you trying to kill me," I ask? "No, Chris, why would you think that?"
Gee, I dunno, Herika. I ask for the solution, and she tells me "snake, eagle, whale" again. It's still wrong. It's still darts. It's still ouch. I tell her again that she's wrong, and she apologies again for her mistake. "What is the correct order then, Chris?" she asks.
Okay. I give up. I tell her the correct order, just so she'll know, and open the gate, stinging not just from my pierced flesh but from the realization that I thought I was paying a bot to help me but it turns out I'm paying to help a bot. I want to be clear: this puzzle isn't a randomized one. The solution to the gate puzzle is always and only snake, snake, whale. So, as a walking walkthrough guide to Skyrim's easiest puzzle, my ChatGPT companion is happy to give useful hints and is calmly apologetic about turning me into a poison-soaked pincushion. Not great at solid answers to direct questions, though, and there's something especially troubling about how confident she is even when she's completely wrong.
Weirdly enough, I also asked Herika about the next puzzle in Bleak Falls Barrow, where there's a dial with three different animals on it, and this time she completely nailed the correct answer. Even stranger, it only took two questions instead of the twenty-five or so I had to ask the first time—and she didn't even ask me if I needed help writing a letter to my mother.
I'm going to continue my adventures with Herika, though I'm still pretty skeptical about trusting her with my life when it comes to providing accurate information. Once we were safely back in Riverwood with the golden claw, I decided to ask her one more time about that first puzzle, which I had eventually given her the answer to.
"Do you remember the puzzle solution from Bleak Falls Barrow?" I asked.
"Yes, I remember it," she said. "The solution was the image of a snake, a whale, and an eagle."
Wrong. Still. Again! At least this time there wasn't a shower of poison darts.
I'll have a lot more to say about using a ChatGPT companion in Skyrim soon: I've already spent hours with Herika as a companion and it's been… interesting, to say the least. If you want to add her to your own game, you can find the mod here, and this (mostly) straightforward tutorial on how to install the mod will help.
