Dwarf Fortress creator Tarn Adams talks about simulating the most complex magic system ever

Forget what you thought you knew about the infamous complexity of Dwarf Fortress. We haven't seen anything yet. Dwarf Fortress hasn't been updated for a year, because developers Tarn and Zach Adams have been preparing it for the most ambitious magic system ever implemented in a videogame. And before magic will come myth generation—simulating the beginning of the world, gods and demons and planes of existence like heaven and hell or Olympus and the underworld. And to prepare for that, they've been rebuilding the game to support sending your dwarves off to the world map for the first time, placing your fortress in the context of a larger, explorable world.

In other words, they've been busy.

I recently spoke with Tarn Adams at the 2017 Game Developer's Conference about his past year's work on Dwarf Fortress. In 2016, we talked about how he was 42 percent of the way towards simulating existence. That percentage hasn't ticked up in the past year, but we had a fascinating hour-and-a-half talk about where he's taking Dwarf Fortress, the difficulty of creating a magic system as deep as he wants, and what that will mean for the game. "We're not afraid to just take the game, pitch it on the ground, and put it back together, because you can't let it impede realization of the vision," he told me.

I've broken our lengthy discussion up into several interviews, each focusing on different aspects of his work on Dwarf Fortress. In this first part, we talk about the magic and myth generation systems coming to Dwarf Fortress in the next two or three years. There's a part about teleportation that's probably going to blow your mind.

PC Gamer: You've been working on Adventure mode a lot, this past year, right? Has that been exclusively?

Tarn Adams: No, no. For the last four or five years, when we add a new sweeping feature—like last year I did that talk about myth generation—so in preparation for that we're trying to hammer down all the details of how artifact type items work, because those are so essential to so many magic systems. We figured we'd take this chunk out, instead of having some three year release delay, that we'll do something that'll take about a year, get that ironed out, and then jump into the generation of the myth systems. We're almost done with that.

I don't want to give my estimate, because it's always wrong, and if it comes out of my mouth it'll be nine times as long. But the artifacts, the way we did that feature, in terms of Adventure vs Fortress mode, first we do the framework and world generation changes. It's been very odd that the last 11 years or whatever, dwarves in your fortress have been able to make 'named objects,' they'd make a named sword if you're lucky or a named floodgate or bucket if you're not, and you can install it in your fortress, do some things with it. They're not that interesting, but they're there, they have names, and historically it tracks them for future games and so forth. But the dwarves in world generation didn't make those things. They were a different sort of culture, which made no sense, there should be items all through world generation.

Now there are, and not just dwarves' [items]. There's religious relics, like the humans, if their high priest of one of their religions dies, they can take a piece of clothing or even a skull or something and put it in a reliquary or whatever and maybe give it a name and venerate it in this temple or whatever. Then someone could come and steal it. Now we have all these world generation stories of stolen artifacts and heroes reclaiming them that we didn't have before, which is great.

If you think, what is the central part of anything that Tolkien wrote, from the Lord of the Rings with the ring popping around, to the Silmarillion with the Silmarils popping around, it's these object things, right? That's not just there, but elsewhere. Harry Potter has these things you have to find. Named objects are very common as a piece of a story.

So that's the first stage, getting world generation done. Then we try to get to them in-play, continue as much world generation stuff as possible. We have been remiss in that. There are a lot of things that happen in world generation that don't get pushed forward, so you have these stories that can't continue, which is bad. So now we have heroes in play that go between different sites, even if you can't see them. They might visit your tavern and say they're doing this or that, but then they're actually going out and doing it, so their stories are all consistent. That we tie into Adventure mode. Adventure mode is like a camera where you can see exactly what's going on in some small area and really diagnose mechanics and how you're interfacing with them. That's why we've been interfacing with adventure mode so much in the past 4-5 months. That's the stage we're at.

Starting, actually next week, is the Fortress mode part of artifacts, which is going to be all new sort of diplomacy things, where invading armies might not just crash against you. They might demand something and they might leave. Or you can send your dwarves, for the first time in 12 years, off the map to reclaim artifacts. And then you have this whole sort of strategy light thing going on, where you have the world map. Finally, your fortress feels like it's part of it. Instead of just, the traders have arrived, the traders leave, the diplomat arrives, the invasion arrives. Now you can send [your dwarves] out. And go to reclaim artifacts that were either stolen by kobolds or something, or maybe weren't yours to begin with, and you can be bad dwarves. Start some shit.

Just starting out, it's not going to be: your squad goes out and then there's this big army battle thing. We're not getting to that yet. There will be some fighting. But you'll have this interaction with the world, and the interactions that point towards you, people sneaking in to steal your artifacts. [They'll target that item] specifically, now, not because it's the most valuable item in your fort. They're going to steal it because this person has a claim on it, with all that information that comes in from world generation. And they can keep forming claims, like you have a human diplomat that arrives, reports back that you have an artifact bucket in your fort or something, and is like 'now that's my personal family treasure' and send someone to steal it from you. And you can send dwarves to go back. We'll have, by the time we're done, all the modes integrated, all the modes worked on, and it'll hopefully bring items up to not quite the status of people, but as the sort of characters they are in these traditional fantasy stories. And then we'll be ready for magic.

When you say all the modes, what does that mean? Adventure and Fortress?

So there's three modes, or four modes, if you think about it. There's the Dwarf mode and the Adventure mode, that's just from the menu. But there's Legends mode which allows you to read the stories. When we talk about world generation, people go back and read these stories. There's a whole class of Dwarf Fortress players who don't play either of the traditional game modes. They just do archaeology on the worlds they've generated. And there's a small cluster who do cartography or set up their D&D game universes based on worlds as inspiration. So having these items, now, that jumps the inspiration level by connecting with everything, you get this level of complexity you didn't have before.

So if I get the timeline right, the first release, which will be fairly soon, will be expanding out to this world map being integrated into the game, where you'll be able to send your dwarves out to other places in the world, and come back from other places in the world instead of just hearing about them?

Yeah, that's right.

What was the next step, after that?

And also in that release is all of this Adventure mode stuff, too. Almost a year [of work], when it comes out, so it'll be chunky. All the artifact stuff. Then there's the bugfixing period, of course, and any small things we can do for them. But then, the talk that I gave last year about creation myth generation, we're going to jump right into that. It's time to do it.

Will that precede magic? Or is magic required for myth generation to work?

You can go about it either way. You can either just say, well what's a bunch of collections of effects? We have a few, like raising the dead, or shooting a fireball or something. There's a few things in the game that are sort of in our proto magic systems. You could come up with a list of effects, but you can't use those unless you generate a magic system of some kind or just put in one magic system like we have with the necromancers. And that's what we want to avoid.

We want to avoid a preset, vanilla list of magic systems that's the same in every game.

So what we're going to do instead is start with the creation myth and have magic be... 'natural' is a weird world to use, but be a natural part of the world that's consistent with those things. In Tolkien it was language and song that had the power of magic, because that's how the creation myth worked. The whole world was sung into existence. Then you had people like Sauron and so forth in the earlier books having these singing duels. I don't know if that was a Finnish thing, there are these warrior poet type things, bard type things, singing stuff. It's consistent, right? There's a deeper connection than just, this is your level one spells, level two spells, level three spells.

So we're going to start with that, and there's going to be, I think, a natural set of spell effects that work well with the myths that get created. There will be some that we don't know how to fit in, but you don't do everything at once. We'll enrich the magic system over time, but there's going to be a natural grouping of say 20-30 effects that we implement first. They'll be different in every world. Some worlds won't have raising the dead at all. Some worlds will have so much dead raising that everyone's practically immortal, or something.

What we want, we don't care about balance, here. We want the world, a lot of its structure, to be determined by what the magical things are about the world. Because that's what makes it a fantasy world simulator, right? It's not all just medieval, medieval, medieval, regular like it is now, with dwarves and elves or whatever. It's something like, if teleportation is easy, that changes everything about the world. That changes economy, diplomacy, espionage, everything.

And you get some of it for free, by the procedural systems, how they work. If someone says 'I want to go here' and then they're like 'oh, I just can,' then it will work that way, and they'll do whatever they were going to do and then say 'I want to go home," and they just do, I didn't have to program anything, except for the teleport effect, and they know that it's linked to moving around.

There are other things, though, like would everyone get together in like, salons, and have some kind of philosophical discussion. 'Can we conceive of a new form of library that involves teleportation?' And that kind of thing. That's the kind of thing we have to teach them how to do. And you want to make sure you don't push too much flavor onto every world that's the same, right? You don't want every world with easy teleportation to have social club teleporters or whatever that was I was just talking about.

It's a balance we haven't had to strike before, or we have and didn't think about too hard, so it'll be interesting to see what comes for free, and that's going to determine our future actions. It's really hard to know, like other emergent behavior in the game, until you start it it's almost impossible to predict.

It sounds like there could be an incredible amount of depth there. The example you gave, you basically threw out a hypothetical example of something really complex you could do with teleportation, that obviously does not exist in the real world. Like how a library would be affected by teleportation…

Well, it's like the internet, kind of.

OK, that works as an analogy. But that's one thing. How does teleportation, if Dwarf Fortress is about simulating existence, how do you think about every aspect of existence that teleportation could affect? How do you go about programming that?

Yeah, that's the problem. It's about telling them what teleportation does, what it's used for, and linking that to their general behaviors, like walking around. The example I gave, where a person that's returning a gemstone to somebody in these new artifact quests or whatever, they know where to get it, they know how to go on a journey, they know teleportation has the mechanic link of 'you can use this to move places.' It's how we do it now when we have the cat licking itself. It knows that action is about cleaning contaminants off their body. That's what it does, so they can interrogate what their action does. We give them helpful hints, so they don't have to read the code themselves. It's just 'if I do this, these things will happen." Then they have in their head that they want to accomplish this goal, and they have a little menu of things that can accomplish it for them. And so hopefully, you never know, teleportation would just be a menu item. 'This is a mover thing, it's an incredibly cheap and effective movement thing.'

But if it were something harder, like, what if the price of teleportation is uncontrolled nausea for a week and you lose a quarter of your blood, or something like that? I don't know how much blood people can live without. But you're just completely out of it for a week or a month. There's still cases where teleport is valuable. So then you need to teach them sort of a cost/benefit analysis type thing. Which, I don't want to be too flippant, but it's not much different than having a different movement value for a forest than a grassland. There's a cost to this movement, and the cost is, 'how much do I value my blood? And how much do I value not being sick all the time?'

Certain people would have different views about that, especially in relation to the urgency of the task. But it's not much different from a regular pathfinding algorithm. You just need to give them access to that tool. 

But with things like the social club, that's a way harder thing to do with a simple goal-oriented utility thing. You need to invent something. At the same time, I'm thinking, well, if these people did want to meet and talk about books, they'd all be teleporting to each other to do it, if it were cheap and easy. Something like that might happen naturally. They wouldn't call it a library, because they wouldn't know how to name that, but they could name their meeting. They have historical events from meeting up. There would just be these meetings happening.

Yeah. So, it's, I'm sure there are ways to put sort of semantic properties on things so that we can teach them to use them a little more freely. And I think that, since we're going to be thinking about this directly when we're working on it, when we have a little menu of effects, I'm hoping we can do some surprising things. I hope my brother and I can do a few things that are really astounding instead of just, like, we added firebolt. That's not interesting.

It seems like a really exciting and dramatically game-changing system you're going to be embarking on. Dwarf Fortress is one of those games where any little thing can have crazy rippling repercussions you wouldn't normally see, but this sounds like an order of magnitude beyond that.

Yeah. I mean, think of something like when we added Z-levels, cross-sectional Z-levels to the game. That changed everything about fortress defenses, fortress layouts, how people thought to make rooms, how fluids worked. Everything changed about the game. We haven't done that very much. There are only a few things that dramatic. This is definitely going to be one of them. But we're not afraid to just take the game, pitch it on the ground, and put it back together. Because you can't let it impede realization of the vision. We're obsessed with getting out to having an actual fantasy world simulator, and storytelling engine, right?

Has how you describe the vision in your own head, has that changed at all over the years? Do you have it in words, now, and is it different than it used to be?

It's not that [different]...we're inspired by a bunch of banal crap, right, like movies we watched as kids. It doesn't mean what we're doing now is childish or even like those things. But still, the general framework is we want to make this schlocky crap, but just have it happen, automatically. At the same time, no, I wasn't planning to do, like—[for example, dwarves] can measure the height of the atmosphere, now.

When we put in the scholarship and knowledge, our generic cutoff year is 1400. And people knew an awful lot by 1400. They knew why rainbows worked with little droplets of water because they had used mirror glass spheres filled with water to make prisms and rainbows. They thought, there's rain and water, they didn't know anything microscopic, but had a really solid theory of after the rain, there's rain droplets in the air that are like these glass spheres we made, and that's why there's a rainbow. They understood it very well.

And every other little thing, they get a little farther than you think. And then we just let the dwarves do it too. That was not an original vision thing. Beastmaster wasn't going around doing science, right? Conan wasn't going around doing science. But we do it. But we still cite stupid crap as our inspiration. It's probably almost disingenuous now, to say those were all our inspirations. It's like, because now we've been to school and know some words or whatever and made a more interesting game than our childhood inspirations. But it still feels kind of like that just because the game is so ridiculous sometimes.

I want to go back to the magic system, because that's exciting to me and probably something you've been thinking about for a really long time. Do you have notebooks full of design ideas of your dream magic system that's been brewing for years and years?

That's the thing, it's never been a dream system, it's been like a dream systematization of what the little nuts and bolts of magic are, so that we can glue them together however we want and surprise ourselves. It always comes back to surprising ourselves. We'd really, really like to have a system like roots, or something, growing into different parts of the game. So integrated that it feels like you're in an exciting and new place. That when they say things like 'oh this magic spell extracts a heavy cost in the future'—which is obviously from Conan the Barbarian movie, always come back to this stupid crap—but that it would actually do that. That if you're making these deals with spirits and stuff, you have made now a social relationship with an otherworldly being. Think of Christian witch theology, deals with the devil to get power, Faust bargain type stuff.

Not just like in say an action-RPG where you're the witch character or whatever, and can curse people and it's like a leveled power you're getting that doesn't change anything about the fact that you have this relationship with some nefarious power, right? But what if you did? And what if they came down and visited sometimes and checked up on you and you could have a conversation in the conversation engine with them, and that was tied into how your magic powers worked? Or if you're one of these little forest spirits and you had to speak to the trees to do things with them? Or even those systems where a lumberjack has to apologize to certain trees that are inhabited by spirits?

If that's just an integrated part of your game, and then you have your dwarves running around, and then they chop down the wrong tree and anger a spirit, that becomes not just a monster spawn, but a character in a game, that you interact with and then that chains into other magical implications.

Magic should feel magical and mysterious, and not necessarily make perfect sense, not necessarily feel like it belongs but feel like it's a part of the setting.
It's just like, you get surprised in Dwarf Fortress the first time when you pour water and it comes up the end of the pipe because we have a pressure simulation. That's what magic should do. Magic should surprise you with how consistent and logical it actually is, but not because you're learning little rulebooks, but just because it feels like it belongs. I'm not sure pressure is immediately obvious to people, or a siphon. It's just kind of weird when you're just blowing into things and they shoot out the other end. It can make sense if you understood it, but certain things you're never going to be able to understand because they're supernatural. You can kind of get at the rules, and kind of not.

These are sort of ideal situations. It's like trying to come up with, there are all these sorts of wizard games, right, where you're doing magical research in a tower or whatever, but you're just getting points. Master of Orion type thing, or Civilization tech tree, getting research points.

But how do you actually model scientific research in a game, that makes it feel like you're experimenting, but it's not as boring as real science? [laughs] Where you have to sit there for six months and see if the two DNA gels have a different line. I worked in a genetics lab for a couple years, and hoo, shit. It's not super exciting. There's just a lot of 'run this 30 times, we're going to look at them, all right, the lines lined up, no outliers, do it again.' So if you get rid of all that stuff, what is the heart of the exciting part of the research? Can we get that and make it feel magical, so you could have some kind of wizard thing like that?

I think this is all still the question where you asked about my dream magic system. It would be one where every system I've described is possible. Some of them might exist in the world, all of them might exist in the world, in which case they might be integrated with each other in subtle ways.

Go back to the Lion, the Witch, and the Wardrobe when Aslan's talking about the deep magic. That there's some underlying thing that could unify those systems and certain other rules. I know the whole thing is a substitute for Christianity, but you know what I mean. Ways to integrate things in layers below the surface. Some of those might not even be accessible to the player, but a lot of them would be over time. And wouldn't that be cool?

And the idea is that it wouldn't be just one way, it would be many ways they could be integrated based on the generation of the world.

Yeah, the creation myth generation, and it just pops out of that. If the forest became alive because the sort of primordial god died and sprinkled its blood on it, but that primordial god has other traits as well, those other traits could come in and influence what forest spirit magic is actually like. And it would be completely unique to that world, and even have a name attached to it through that god, so the societies of whatever people end up doing things related to that, might pray to that god, be afraid of that god, XYZ, you can do anything with that.

And then you can inflect the character different ways based on these other properties. But then you could have some mythological event that happened that flips that on its head. You can have events later in the game when the player's actually playing their third fortress or something, they might accidentally trigger something that flips it back, or something, and changes the rules. And suddenly people's teleport spells don't work anymore, and they'd have to be able to deal with that. All these social clubs break down because no one can visit each other anymore.

I hadn't even thought of the idea of the magic system being changeable within the simulation. 

Yeah!

I just thought about generating the world and X magic exists within it.

That's one of our primary checkboxes. I think it's on the development list. It's on ours, if it's not on the public one. Check out the dev page. But it's just 'change.'

These things can change.

Right after the magic system we're doing the property/law/customs framework.

To set up: why is your fortress here and what is its relationship to the larger world? In those senses we have no conception of it right now. But law, unless it's some kind of divine law or something, is mutable. Very mutable. People change it all the time. That's a similar principle as these magical things.

There could be correspondences between them, if it's one of those celestial bureaucracy things. Whatever the term is for troubles in the heavens lead to troubles below, troubles below can lead to troubles above. There can be correspondences between the law systems and tyrants and what's going on, and that could turn off a magic system if someone's throwing a fit up there, suddenly magic doesn't work.

Like the greeks, all the time.

Yeah, they could just be characters, do things, take things away, be…

Dicks?

Dicks. Dicks was the word I was searching for an alternative for. It wasn't 'whimsical,' that's a little too happy. 'Capricious.' Some word like that. Now imagine you had just four hours to sit and think about things, you can milk all kinds of weird ideas out of this. Then you think, what's the implementation order? Ha ha ha ha. I have no idea what I'm getting into. So you don't want to promise a specific thing is going to be in there, but you just say, there's going to be a lot of cool stuff in there. It just keeps rolling out of my mouth, it's really interesting what you can do with this thing.

I know there are people who have expressed concerns about this totally screwing up the game. Like I said, just throwing it on the ground. And there are ways to mitigate changes to the game, even if you have to just say, 'there are options to roll things back and most of the time maybe dwarves themselves are not really that magical.' Which is not unusual, right? Usually dwarves are at most given runes to stick on items or do geomancy type stuff. They're not always super magical.

We've been so fortunate to have picked dwarves.

Was that a totally random choice?

It was this random: We had the Mutant Miner game. It started as this side project. We had trouble adding multiple miners, it was turn-based, it was just a mess. So we were like, we need to make this game real time, it's got a bunch of miners running around, we kind of want to change the setting. What is a place where you'd have a bunch of miners running around in a different setting? Dwarves!

And so now we have dwarves. But dwarves brought so much more to the table. The modern conception of a fantasy generic dwarf is they're kinda drunken, they have flaws, they can be too boistrous, so bug reports are often treated as just funny. Dwarfy is the word that's thrown around. Something dwarfy is going on. So similarly, they give us this whole new power of, well, the procedural magic system could be rolled back as much as we need to, either temporarily or permanently in some cases, for fortress dwarves, which keeps the core gameplay intact while having all kinds of cool stuff going on in the larger world and you can just play in adventure if you want to mess things up.

At the same time, if a wizard is coming from outside to visit your fortress and has something very disruptive, that's your problem.

Kill him.

The things people were concerned about is, well what if I use my library to set up dwarves doing magical research, and then I just roll some horrible thing and it's 10% chance your fortress blows up or something. If that's how magic works. You want to have magical disasters, but if the magical disaster has in a fort you've been running for 10 years because of some coin flip, not quite as desirable. You're telling people to roll with a pretty big punch. The magic disaster has to at least be something you can see unfolding in your fortress. It has to be entertaining. The losing is fun. It better be fun. It shouldn't feel like a crash bug, just goes [snaps finger].

We had problems like that. We always got rid of them. Like the demons. Well, spoiler alert, but it's been many years. The demons that can take over your fortress, it used to not even, there wasn't even a battle or anything. It just says 'your fortress has fallen into ruin.' Pops up a black screen. You were supposed to go in adventure mode and investigate and then you find the demons, but that's a tall order. Especially if people don't like both modes.
Yeah. We're not assuming a lot of risk, I think. There's just the risk of doing something that could take more than a year, again.

So you're almost finished with artifacts. You're then going to focus on magic, or integrating the creation system first? Or simultaneously?

We're trying to do one release, and it may not have the breadth of magical effects, but it has point A to point Z with creation myths, which we already have a generator for that's pretty varied, and then magic systems, and magical objects. There's magical landforms. The creation myth is just flippantly throwing around magical floating mountains and all kinds of other stuff, but none of that's supported in the game. So you have to be careful with your landforms. We want to support at least one or two types of magical landforms, one or two types of planar things.

We don't know how far we're going to get with that. That's really big. That's like the Z-level, adding a planar system where you can go to a different place is a huge grind, basically, of support. For basically a fourth variable. It's something we really want to do, though. Then support four or five types of magic systems, means of integration with the other elements within the game.

And then build from there?

Yeah. Release, and kind of decide: do we do another three or four month essentially expansion pack of that, or do we do the law/property/custom thing and switch? And then it just goes back into the category of everything else in the game. Stuff is slowly added to everything, but it's not a concerted push anymore.

But I have a feeling that the concerted push on magic could last a few releases. Especially if it works really well, we'll be interested in doing three or four more pushes on it. So if the whole process lasts two or three years, I wouldn't be surprised. And I'd hope to have more than one release in that 2-3 year window, to keep it interesting and keep people  realizing the potential once they see the potential is there. Then we can do more.

We're also not quite as excited, but would be happy to get to the property stuff too, because that's been missing for so long. We'd be able to re-enable the economy and have trade things going on, which is important, because the world seems dead on that axis.

We brought war halfway up and running and now we've got these items running around, so there's all this interpersonal related item stuff going on, but there's no property, economic stuff going on. It would be nice. It also gets us to boats, which we want to do. Boats are super cool.

When you mentioned the planar system, is that planes of reality, like heaven, Olympus…

Yeah, there's lots of different ways to consider it. There's the D&D way of doing it. You have different elemental areas you can go to, but also different heaven/hell type areas you can go to, that are all interlinked with different mechanics. The planes as layers, but in D&D there's astral and ethereal connections, stuff like that. There's mechanics, places you can be. Then in the Tolkien stuff, there's the shadow idea, where it's the same place, but it's not just becoming invisible, you're going into this whole different world.

The Japanese stuff in that new Nioh game. I just got through it. The monsters would create areas of this yokai realm. It's a different place, but it exists parallel or on top of the existing world, so there's different ways to consider them.

There's also meditative stuff, where you can just sit and meditate, and move out of your body and be in a place where only spiritual things are, but your body still exists. All of those can work in a similar framework.

Basically the problem is having two places loaded at once that are not just contiguous parts of one region. That is exactly the same as being able to see your dwarves recovering the artifact when they leave the fortress. Just pretend it's a different plane of existence. Then you have two play areas, and I don't know, I'm obviously a big interface expert, so you could have tabs at the top or something to switch between the different views. You have to be careful about optimization, because the game's already an optimization problem. But the secondary places we load wouldn't have a whole other running fortress. They'd be simpler to avoid killing the processor.

It was cool to sit and think of how those things were similar so we can do them all at the same time. Not necessarily complete, some kind of army thing at the same time, but make sure that when we code planes we're supporting dwarf squads being visible off-map. And any other thing that you might think of that involves two different areas being visible at once.

You could really serve a lot of purposes.

It's like the Z coordinate slices. Another bashing of the game. It's a hard thing to decide to do that all at once. Do the magic systems, do this bash. It takes time, but the rewards on the other side seem so worth it.