Path patching / IOI circuit

Mechanistic family · patching the wiring, not the nodes

Learn the method — a beginner's guide

New to explainable AI? This block explains the method as it is used on neural networks, then how we run the very same method on an Atari 2600, where — unlike for a network — we know the right answer. Every implementation step links to the code. See also the XAI primer for the shared vocabulary.

The idea in plain words

Activation patching tells you that a component matters. It does not tell you how its influence reaches the output. Component A might act on the output directly, or it might only matter because it changes component B, which then changes the output. A circuit is the answer to that question: the small set of components and the edges (connections) between them that carry out a behaviour. To find edges you need a sharper tool.

Path patching is that tool. You change a sender component, but you let the change reach only one chosen receiver. Every other component downstream of the sender is held at the value it had in the normal run. If the output still moves, the signal must have travelled along the sender→receiver edge. This is how Wang et al. mapped the indirect-object-identification (IOI) circuit in GPT-2: which attention heads send which information to which other heads to complete “When Mary and John went to the store, John gave a drink to…” with “Mary”. On the VCS the true wiring between memory cells can be measured exactly, so we can check how much of it a path patch actually recovers.

How it works on a neural network

Think of the network as a directed graph of components. A transformer’s components all read from and write to the residual stream, so every earlier head can influence every later one. Pick a sender \(s\) and a receiver \(r\). Path patching runs the clean input \(x\) with three rules: \(s\) takes its value from the corrupted input \(\tilde x\); the receiver \(r\) is recomputed normally, so it sees the corrupted \(s\); and every other component that \(s\) could reach is frozen to its clean value. Then

\[ \Delta_{s \to r} \;=\; y\big(x \,\big|\, s := s(\tilde x),\; \text{others frozen},\; r \text{ recomputed}\big) \;-\; y(x). \]

A non-zero \(\Delta_{s\to r}\) means the edge \(s \to r\) carries part of the behaviour. Repeating this over pairs, working backwards from the output, builds the circuit. The result is scored as a set of edges.

Toy example. Let \(b = 2a\), \(c = a + b\), and output \(y = c\). Clean \(a = 1\): \(b = 2\), \(c = 3\). Corrupt \(a := 2\). Plain activation patching of \(a\) gives \(b = 4\), \(c = 6\), so \(\Delta_a = 3\). Now path-patch the edge \(a \to b\): let \(b\) see \(a = 2\) (so \(b = 4\)), but freeze \(a\)’s direct input to \(c\) at its clean value 1. Then \(c = 1 + 4 = 5\) and \(\Delta_{a\to b} = 2\). Path-patch the direct edge \(a \to c\) instead: \(b\) stays at 2, \(c = 2 + 2 = 4\), \(\Delta_{a\to c} = 1\). The total effect of 3 splits into an indirect part (2, via \(b\)) and a direct part (1). That split is what node-level patching cannot give you.

From a neural network to the Atari

A component (head)	A candidate RAM cell: a byte the game uses as a variable.
Layers	Video frames. The sender lives at frame \(f^*\), the receiver at frame \(f^*{+}1\).
Sender corruption	Write base+17 into the sender cell at \(f^*\).
Freeze all other components	After one frame, overwrite every other candidate cell with its clean value at \(f^*{+}1\); only the receiver keeps the sender’s signal.
Metric	Does the receiver cell’s own value differ from the clean run at the end of the 15-frame horizon?
Reference circuit	The true routine: the same path patch with an exhaustive set of sender values (0, base+17, base+37). An edge is true if any value fires it.

How we apply it to the VCS

The nodes are the game’s candidate RAM cells. Everything starts from the shared gameplay checkpoint at \(f^*\) with a 15-frame horizon. The first frame of the horizon is the sender→receiver hop, using the gameplay stream’s own next action.

1. One path patch \(i \to j\): copy the checkpoint, write the sender value into cell \(i\), step one frame, then restore every candidate cell except \(j\) to its clean frame-\(f^*{+}1\) value. Run the remaining 14 frames. The edge fires if cell \(j\) ends up different from the clean run.
2. Recovered circuit. One sender value per edge (base+17), the classical single-shot path patch, over all ordered pairs of cells.
3. True routine. The same edge-restricted operation, but with three sender values. This is the faithful direct circuit the method targets. A separate transitive graph (perturb a cell, run the full horizon, see which cells change) is recorded for context only.
4. Score. Edge precision, recall and F1 of the recovered circuit against the true routine, with self-edges excluded.
5. Positive control. Re-deriving the circuit with the exhaustive value set must give F1 = 1, proving the scorer rewards a faithful circuit.

Two differences from the transformer original matter. First, our “layer” is a whole frame of thousands of CPU instructions, and a cell can be read and rewritten many times inside it. Second, we freeze only the candidate cells, not every byte, CPU register or TIA register. A signal can also leak through those unfrozen parts. The runner also counts transient receivers: cells the program recomputes from scratch, so no single-frame path into them can survive to the end of the horizon.

Read the code

XAI the interpretability method   substrate the bit-exact emulator it runs on (jutari, Julia).

• XAIpath_patching.jl:310 — One path patch: corrupt the sender, step one frame, freeze every other candidate cell, run the rest.
• XAIpath_patching.jl:377 — The method: single-shot path patch with one sender value per edge.
• XAIpath_patching.jl:353 — The reference: the same edge-restricted patch with the exhaustive value set.
• XAIpath_patching.jl:263 — The transitive (multi-hop) data-flow graph, recorded for context.
• XAIpath_patching.jl:424 — Circuit precision, recall and F1 over off-diagonal edges.
• XAIpath_patching.jl:493 — The sender→receiver hop uses the gameplay stream’s next action.
• substratejutari_oracle.jl:176 — The byte write used for both the sender corruption and the freeze-restore.
• substrateshared_testbed_impl.jl:115 — The shared gameplay checkpoint and action stream.
• substrateStellaEnvironment.jl:259 — One bit-exact frame: the one-frame hop and the remaining horizon.
• substrateoracle_intervene.jl:163 — Reads the candidate cells that become the circuit’s nodes.

Reading the result

Each game’s figure compares two adjacency matrices: the recovered edges and the true routine. Expect low recall on most games. A single sender value often fails to trip an edge that another value would. Many cells are also rebuilt by the program within the frame through other cells, so the one-step path carries no surviving signal. On games where the direct circuit is genuinely empty the method trivially matches it. On others, including the Pong state shown here, it can recover almost nothing. The positive control shows this is a real limit of the single-shot method, not a scoring bug.

Pitfalls

• Which nodes you freeze defines the answer. Freezing too few lets the signal leak around the chosen path; freezing too many can break the computation. On the VCS we freeze only candidate cells.
• One corruption value is not enough. An edge can carry a signal only for some values (a branch that triggers above a threshold). Single-shot recall is limited by that.
• Coarse layers. A VCS frame contains many reads and writes of the same cell. A one-frame hop merges several real edges into one, or hides them.
• Metric fragility. Miller et al. show that circuit faithfulness scores in transformers change a lot with the ablation type and the metric. Report your choices.
• Cost grows with pairs. One re-run per sender–receiver pair and value: quadratic in the number of nodes. This is why ACDC and edge attribution patching exist.

Try it yourself

julia --project=jutari tools/xai_study/phaseC_mechanistic/path_patching.jl --games core

Writes path_patching_<game>.json/.npz and a core summary into tools/xai_study/phaseC_mechanistic/out/. The exhaustive reference needs up to three re-runs per ordered cell pair, so this is the slowest of the patching runners; it can be sharded per game on the cluster (--shard i --nshards n).

Original papers & further reading

• Wang, Variengien, Conmy, Shlegeris & Steinhardt (2022). Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small. ICLR 2023. — arxiv.org/abs/2211.00593
• Goldowsky-Dill, MacLeod, Sato & Arora (2023). Localizing Model Behavior with Path Patching. — arxiv.org/abs/2304.05969
• Miller, Chughtai & Saunders (2024). Transformer Circuit Faithfulness Metrics are not Robust. (Shows that circuit scores depend heavily on ablation choices.) — arxiv.org/abs/2407.08734

Read next: Activation patching / causal mediation · ACDC — automatic circuit discovery · Attribution / edge patching · A1 · Connectomics / data-flow graph