Interchange interventions / DAS

Mechanistic family · testing a hypothesis about a variable

Learn the method — a beginner's guide

New to explainable AI? This block explains the method as it is used on neural networks, then how we run the very same method on an Atari 2600, where — unlike for a network — we know the right answer. Every implementation step links to the code. See also the XAI primer for the shared vocabulary.

The idea in plain words

Suppose you believe a network solves a task with a simple algorithm. For example, to compare two numbers it first computes “which one is bigger” and then reports it. That simple algorithm is a high-level causal model: a few named variables and the arrows between them. The question is whether the network really contains a variable that plays the role of “which one is bigger”. Causal abstraction answers it with an interchange intervention: take the hidden state that should hold the variable from a source input, put it into the run on a base input, and check whether the output changes exactly as the high-level model predicts when you swap that variable.

The catch is that a network rarely stores a variable in one neuron. It may be spread across many neurons, along some direction of the activation space. That is a distributed representation. Distributed Alignment Search (DAS) therefore learns a rotation of the activation space. After rotating, the first few coordinates should hold the variable, and swapping only those coordinates should behave like swapping the variable. On the VCS we already know where every variable lives, so the method turns into a clean test of the swap logic itself, and a lesson about what a perfect score can and cannot prove.

How it works on a neural network

Let \(h(x) \in \mathbb{R}^d\) be the hidden vector at some layer for input \(x\). DAS learns an orthogonal matrix \(R\) (a rotation: \(R^\top R = I\), so no information is lost) and fixes a number \(k\) of coordinates. The interchange intervention on base \(b\) with source \(s\) is

\[ \hat h \;=\; R^\top\!\Big( (I - P_k)\,R\,h(b) \;+\; P_k\,R\,h(s) \Big), \]

where \(P_k\) keeps only the first \(k\) rotated coordinates. The rest of the network then runs on \(\hat h\). The high-level model predicts an output \(y^\ast(b, s)\): the answer you would get on \(b\) if its variable took \(s\)’s value. \(R\) is trained by gradient descent so the network’s output matches \(y^\ast\) on many \((b, s)\) pairs. The score is interchange intervention accuracy (IIA): the fraction of held-out pairs where the swapped run gives \(y^\ast\).

Toy example. Let \(h = (h_1, h_2)\) with \(h_1 = u + v\) and \(h_2 = u - v\), where the variable we care about is \(u\). No single neuron is \(u\). But rotate by 45°: \(R h = \tfrac{1}{\sqrt 2}(h_1 + h_2,\; h_1 - h_2) = \sqrt 2\,(u, v)\). Now the first rotated coordinate is exactly \(u\) (times a constant). Base \(u{=}1, v{=}3\); source \(u{=}5, v{=}0\). Swap coordinate 1: the patched state encodes \(u{=}5, v{=}3\), so \(h = (8, 2)\). That is precisely the prediction “base with source’s \(u\)”. Swapping neuron \(h_1\) alone would give \(h = (5, -2)\), which decodes to \(u{=}1.5, v{=}3.5\): the wrong \(u\) and a corrupted \(v\). This is why DAS looks for a direction, not a neuron.

From a neural network to the Atari

Hidden vector \(h\) at a layer	The VCS state at frame \(f^*\): 128 bytes of RAM plus TIA registers.
High-level variable \(V\)	A named game variable from the candidate list (a position, a score digit, a timer), plus the background colour.
Learned rotation \(R\) and subspace \(P_k\)	Not learned here. The aligned “subspace” is one axis-aligned byte: the program stores each variable in its own RAM cell.
Base / source inputs	Base: the clean gameplay stream. Source: either a directed value (base+17) or a natural one (the same stream with RIGHT pressed before \(f^*\)).
IIA	Fraction of variables whose cell, after the swap, holds the source value the high-level model predicts.
Wrong-subspace control	Swap the next candidate cell instead and check variable \(V\): accuracy should collapse.

How we apply it to the VCS

The runner reads its variables from the game’s candidate file: each candidate RAM byte becomes a variable whose value is the byte itself, plus a background-colour variable read from a screen pixel. The claim being tested is that “variable \(V\) lives in byte \(i\)”. Everything runs on the shared gameplay checkpoint at \(f^*\), with a 15-frame tail.

1. Interchange-effect matrix. For every cell \(i\), write the directed source value base+17 into a copy of the base checkpoint, run 15 frames, and read every variable \(j\). The same swap is replayed from a fresh boot; the two matrices must be identical.
2. IIA of the true alignment. For each RAM variable, swap its own cell and read the cell back at the swap frame. It must hold the source value. This is done for the directed source and, where the natural RIGHT-press source really differs from base, for that too.
3. Misaligned control. Swap the next candidate cell in the list and read variable \(V\). It matches the prediction only if the two cells happen to alias, so accuracy should drop to near zero. This is what makes a perfect aligned score meaningful.
4. Alignment search. For each variable, the runner picks the cell whose swap best reproduces the variable’s own 15-frame effect, and checks whether that is the true cell. Cells the program overwrites within the horizon (transient cells) give no signal and are listed separately.

Be clear about the difference from real DAS. There is no learned rotation and no gradient training. The search space is the set of single bytes, because on the VCS the program’s variables are stored one per byte. The concept labels come from the candidate file, not from the method. So the VCS run checks that interchange logic and the IIA metric reward the right alignment and punish a wrong one; it does not show that DAS would discover the variable unaided.

Read the code

XAI the interpretability method   substrate the bit-exact emulator it runs on (jutari, Julia).

• XAIdas.jl:227 — The high-level variables: one per labelled candidate byte, plus background colour.
• XAIdas.jl:250 — The interchange intervention: write source values into a copy of the base checkpoint, run the tail.
• XAIdas.jl:337 — The natural source run: same gameplay prefix, RIGHT pressed before \(f^*\).
• XAIdas.jl:408 — The directed source values (base+17), so every cell gets a real swap.
• XAIdas.jl:447 — IIA readout: does the variable hold the source value right after the swap?
• XAIdas.jl:476 — The misaligned negative control: swap the wrong cell, read the variable.
• substratejutari_oracle.jl:176 — The byte write that realises every interchange.
• substrateshared_testbed_impl.jl:115 — The shared base state every Phase C method sits on.
• substrateStellaEnvironment.jl:259 — One bit-exact frame; the 15-frame tail is a loop over it.
• substrateoracle_intervene.jl:242 — The oracle. A single-cell interchange is the same operation as its do(cell := value).

Reading the result

Expect the aligned IIA at the ceiling and the misaligned control near the floor. The gap between the two is the real content: it shows the metric discriminates. Then look at the alignment search and the transient list. Variables whose cell is rewritten every frame cannot be aligned by their downstream effect, because there is none within the horizon. A perfect score here verifies a given label; it is not evidence that the method found the label itself.

Pitfalls

• A supplied variable is not a discovered one. DAS tests an alignment you propose. If you choose the hypothesis from the same labels you score against, a perfect IIA is guaranteed in advance.
• Interpretability illusions. Makelov et al. show that a learned subspace can reach high IIA by activating a dormant pathway the model never uses on real inputs. Always compare against a control and check the subspace on natural data.
• Too much freedom. A large subspace or a very expressive map can make almost any hypothesis “fit”. Keep \(k\) small and report held-out accuracy.
• Readout timing. On the VCS, reading the variable right at the swap is trivially exact. Reading after 15 frames shows that many cells are overwritten by the program, so present is not the same as used.

Try it yourself

julia --project=jutari tools/xai_study/phaseC_mechanistic/das.jl --games core

Writes das_<game>.json/.npz and das_core_summary.json into tools/xai_study/phaseC_mechanistic/out/. One re-run (plus one fresh-boot replay) per candidate cell; minutes per game on one CPU core.

Original papers & further reading

• Geiger, Lu, Icard & Potts (2021). Causal Abstractions of Neural Networks. NeurIPS. — arxiv.org/abs/2106.02997
• Geiger, Wu, Potts, Icard & Goodman (2023). Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations (DAS). CLeaR 2024. — arxiv.org/abs/2303.02536
• Makelov, Lange & Nanda (2023). Is This the Subspace You Are Looking For? An Interpretability Illusion for Subspace Activation Patching. ICLR 2024. — arxiv.org/abs/2311.17030

Read next: Activation patching / causal mediation · Causal scrubbing · Linear probing + control tasks · Sparse autoencoders