Notes on Mechinterp with DELTARUNE

June 17, 2026

And so wept the fallen star, making rivers with its tears.

obj_dw_churchb_bookshelf

I recently watched a video essay by Andrew Cunningham that suggests that motifs of DELTARUNE form a “symbolic network.”

[The symbolic network is] a constellation of words, images, and concepts echoing throughout the game with conspicuous emphasis, but scant little in the way of explanation.

Cunningham

Cunningham maps in extraordinary detail nine particular motifs (heaven, hell, light, hope, water, darkness, stars, glass, prophecy), and what the connections between them imply. He asserts, for example, that “water” and “darkness” function as near-aliases in the game’s cosmology.

A web of concepts is, more or less, the thing a language model is trained to build. Which had me thinking, what would the game’s network look like from the perspective of a language model? To answer this, I finetuned a model on the game’s dialogue. I hope you’ll find something interesting.

Approach

I took Pythia-160m, a small open-source language model from EleutherAI (12 layers, 768-dimensional hidden states, ~160 million parameters), and finetuned it on all the text from the game which comprises roughly 71,000 lines of dialogue across Chapters 1–4, which is everything at the time of writing. I trained it for 5 epochs with a learning rate of $5 \times 10^{-5}$ and a context window of 512 tokens.¹

I picked Pythia because it was designed for interpretability work, which is the thing I’m doing right now. It’s also small enough to overfit on DELTARUNE’s text, and while overfitting is typically not a “good thing,” forcing the model to fit this corpus specifically is, actually, exactly what I want.

I then picked some words from Cunningham’s essay (plus some more just for fun). If you’ve played DELTARUNE, I hope you’ll agree that these words are “thematically loaded” in the context of the game:

I saved the model weights from before and after finetuning, so each experiment can compare what the model “knew” about these words in English, versus what it knows after reading the text dump.

Does Anything Happen

The first test I thought to try was to give the finetuned model the beginning of a sentence that might plausibly invoke a motif and compare its response to that of the base model. Here’s what we get…

Huh, that’s interesting! “The shadow of the fallen” now completes to “angel” almost 57% of the time, where the base model’s top pick was “tree”. Similarly, the model thinks a hymn echoes through the “darkness” rather than a “church”. You can even pin down which word does it. For example, with “The shadow of the fallen,” we can ask, with input attribution, how much each word in the prompt pushed the model toward “angel”. We do this by computing the gradient of the output logit (which is the model’s raw score for a word, before it’s turned into a probability) for angel with respect to each token’s embedding, multiplying by the embedding elementwise, and taking the norm.

“Fallen” is 60% responsible for the model’s next word being “angel”.

Methodology: for each input token $x_i$, compute $\|\nabla_{x_i} \ell_{\text{target}} \odot x_i\|$ and normalize across the sequence.

What is a Feature

So, the model learns associations between words from training. We might well ask how it does this.

In May 2024, researchers at Anthropic released a paper (read the associated article here, it’s really good!!) demonstrating that they could extract “features” from their model. Okay, so what is a “feature”?

At each layer in a model, its “hidden state” is just a vector. Imagine a simple model where these vectors are three numbers long:

This might be pictured as an arrow pointing in some direction in space. The same way we calculate the distance between two points in 2D with $\sqrt{(x_2-x_1)^2 + (y_2-y_1)^2}$, and in 3D with an extra $z$ term: $\sqrt{(x_2-x_1)^2 + (y_2-y_1)^2 + (z_2-z_1)^2}$, nothing stops us from adding more terms. Our model’s hidden states are 768 numbers long…

The distance betweeen two of our 768-number vectors $\mathbf{a}$ and $\mathbf{b}$, just uses the same formula with 768 terms:

…so for our purposes this is still a “direction.”

Anthropic found that there were specific directions in this space that lined up with human concepts. For example, they found one for the Golden Gate Bridge, another for insecure code, another for sycophancy, and so on. When they turned the Golden Gate direction up, the model couldn’t stop bringing up the bridge everywhere.

A feature is one such direction $\mathbf{f}$. How much a given input triggers it can be measured. If that measurement is near zero, it means the concept is absent and vice-versa. There are two things you can do with a direction like this. You can add it to the model’s hidden state, which turns the concept up; this is called steering. Or you can subtract it, this is called ablation. (Ablation is pretty cool too! Arditi et al. found that models refusing to do tasks could be steered by a vector representing refusal, and ablating that direction stops the model from refusing to do things.)

Anyways, I built a direction for “motif-ness” by taking the average hidden state (at layer 8) across motif-rich DELTARUNE text, subtracting the average across generic English sentences, and normalizing. (This is a difference in means, not a proper sparse “feature” of the kind Anthropic extracts.²) This yields a single 768-dimensional steering vector pointing toward DELTARUNE motifs. And you can turn $\alpha$ up!!!

Cranking $\alpha$ Up

When generating text, I add $\alpha \cdot \vec{v}_{\text{motif}}$ to the hidden state at every token position. The more I crank $\alpha$ up, the more the model’s responses change. Here’s the model continuing Bible passages as $\alpha$ climbs:

Methodology: Get a steering vector $\vec{v}$ with $\vec{v} = \frac{\bar{h}_{\text{motif}} - \bar{h}_{\text{general}}}{\|\bar{h}_{\text{motif}} - \bar{h}_{\text{general}}\|}$, where $\bar{h}$ is the average hidden state at layer 8 averaged across all token positions and all sentences in each set. During generation, forward hook on layer 8 that adds $\alpha \cdot \vec{v}$ to every token’s hidden state at every step.

There’s recent work suggesting some concepts live on curved surfaces (manifolds) that straight lines only approximate. On these, you would steer along the surface rather than in one direction. Goodfire did some research on this earlier this year. I did try to build a motif manifold myself but couldn’t get it to work, maybe because I don’t have enough time on the GPUs so you’ll have to accept my linear vector for now.

Is Water Equivalent to Darkness?

Cunningham says that in the symbolic language of DELTARUNE “water” and “darkness” work as near-aliases. So I tried to check that directly. Maybe they should converge after finetuning more than unrelated words do. I measured the cosine similarity (instead of how far apart two vectors are, how closely they point the same way, from −1 to 1) between the average hidden state (layer 8) for water, darkness, and a couple other comparison words, both pre and post-finetuning:

They don’t converge. Actually, everything drifts apart a little. This is probably because finetuning on such a specialized corpus makes the representations more specialized, which pulls every pair apart, and so we see water–darkness ($\Delta = -0.028$) drift about as much as water–forest ($\Delta = -0.031$).

This was disappointing, but then I realized that “water” shows up in hundreds of lines, only a handful of which are really meaningful motif-wise, and the signal might be getting drowned out. So I don’t think this exactly refutes the essay, especially because in DELTARUNE (because it’s a VIDEOGAME) motifs can lie in the music and visuals, and there might be more there the model just can’t pick up on.

W. D. Gaster

Steering isn’t unique to “motifs,” you can actually extract a vector for any concept. Speaking of which…

W. D. Gaster is a character in DELTARUNE.³ Or maybe, outside of it…? It’s unclear. In any case, the DEVICE_* dialogue lines from the vessel creation sequence, game-over screen, save menu, etc. are attributed to him and we can run this same technique on his lines.

Using the same method I used for motifs (taking the mean hidden state at layer 8 across all of Gaster’s 31 lines of dialogue, subtracting the mean across 30 lines of general DELTARUNE dialogue lines, and normalizing) I got a single $\mathbb{R}^{768}$ steering vector that points towards “Gaster-ness.” The most Gaster lines in the game are:

Using the Gaster direction as a steering vector and cranking up $\alpha$, we get:

The motif vector, at high $\alpha$, produced text about water, darkness, light, prophecy, fountains, hope. The Gaster vector produces different text (“PREPARATIONS ARE COMPLETE”), which makes sense for a character who was once the Royal Scientist before he got shattered across time and space. This is probably why I later found that the English words the Gaster direction is closest to in the model are “nucleotide,” “systematically,” “yield,” “correspondence,” “governing,” and “finances.” All very formal.

And, this probably doesn’t mean anything, but I appreciate that though none of Gaster’s dialogue in DELTARUNE has anything to do with hands, a high Gaster $\alpha$ prompts the model to generate “The hand that… The hand that…”

Conclusion

A language model (even a small one) finetuned on DELTARUNE’s dialogue learns some cool associations. To be honest, I’m still not sure what to make of any of this. But I AM excited for Chapter 5. See you later!