An attention map shows where a head looks. It does not show what the head moves. Take the deeper pointer model from the last post and ask two questions. Does it compute the right thing? And can you see how by reading its attention? The answers come apart, and that gap is the lesson of mechanistic interpretability in miniature.
Behaviorally, it is a perfect pointer
There is a test that does not depend on reading any attention map. Flip the bit at the addressed memory cell and check the output: it tracks the new value about 99.5% of the time. Flip any other cell and the output changes about 0.5% of the time. So the model reads exactly the addressed cell and nothing else. That is airtight, and it required looking at no internals at all.
Mechanically, the clean circuit is gone
At small memory the attention was legible. One layer gathered the address, the next spiked on the addressed cell, and you could read the dereference straight off the maps. Scale the memory up and that legibility evaporates: the final query barely concentrates on the correct cell for most addresses. The flip test says the lookup is happening anyway. So which is it?
The resolution: attention weight is not information flow. A head’s output is the attention-weighted sum of values, and the value projection can quietly suppress a cell you attend to hard, or carry real information through a cell you barely attend to at all. And the computation spreads across heads and layers. The attention map is a weak instrument the moment the work distributes.
The causal probe that recovers it
The right tool is causal. Take a clean run and a run with the addressed bit flipped. Then copy the clean run’s residual stream (the vector each position carries from layer to layer) into the flipped run, one (layer, position) pair at a time, and see where restoring it restores the answer. A clear picture appears that the attention maps hid: the bit’s value is transported from the addressed cell toward the readout position, across the layers. The structure was there the whole time, in the residual stream, not in the attention weights.
What actually happened, including the wrong turns
The tidy version of this story is a lie, so here is how it went. I started with a clean hypothesis: that a particular layer “refines” the lookup the earlier layers begin. The data refuted it. Second hypothesis: the heads partition the address space among themselves. The data refuted that too. What survived is narrower and less satisfying: the model computes the right cell (the flip test is not negotiable), the extra depth buys a multi-stage assembly of the query, and the dereference itself is distributed in a way the attention maps will not show you. Two nice stories died to get there.
This is the induction-head story, small
None of this is exotic. It is the induction-head and in-context-learning picture compressed into a toy. In real language models the same primitive, find the relevant earlier position and copy what is useful from it, is spread across many heads, and ablating any single one barely moves the behavior because the others cover for it. The clean mechanistic accounts in that literature come from causal patching, not from looking at attention. Going from small memory to large here is that arc in fast-forward: a legible circuit when the problem is tiny, illegible attention when it grows, and a causal probe that brings the mechanism back.
Behavioral and causal tests can stay airtight while reading the mechanism off the attention weights gets worse with scale. That is why the field reaches for causal interventions, and why “head 7 is the lookup head” is a sentence you should not believe without a causal test behind it.
The flip test, the per-layer causal trace, the comparison to induction heads, and the two refuted hypotheses written up in full are in Chapter 7 of the book, Inductive Biases in Neural Networks.
Watch the Episode
This post is also a video episode: Attention Weight Is Not Information Flow, part of the series playlist.
Discussion