Causal scrubbing

Mechanistic family · testing a circuit hypothesis

Learn the method — a beginner's guide

New to explainable AI? This block explains the method as it is used on neural networks, then how we run the very same method on an Atari 2600, where — unlike for a network — we know the right answer. Every implementation step links to the code. See also the XAI primer for the shared vocabulary.

The idea in plain words

Most interpretability methods produce explanations. Causal scrubbing checks one. You bring a hypothesis: “for this behaviour, only these parts of the network matter, and they compute this”. Causal scrubbing turns that sentence into an experiment that the hypothesis can fail.

The trick is a simple consequence of the claim. If a part really does not matter, then replacing its activation with the activation it had on a different, random input should not change the behaviour. So: take everything the hypothesis calls irrelevant, resample it from other inputs (this is the “scrubbing”), run the model, and measure how much of the original performance survives. If it all survives, the hypothesis passes. If performance collapses, something the hypothesis called irrelevant was in fact doing work. It tests a whole explanation at once, not one component at a time, which makes it one of the strictest checks in mechanistic interpretability.

How it works on a neural network

A hypothesis has three parts: the model's computational graph \(G\), a simpler “story” graph \(I\), and a map \(c\) saying which nodes of \(G\) implement which nodes of \(I\). Scrubbing rewrites the forward pass on input \(x\). Every activation not claimed by the hypothesis is taken from a random other input \(x'\). Every activation that is claimed may be taken from any input \(x''\) that agrees with \(x\) on the feature the story says it computes. The score is the share of the loss gap that survives:

\[ \text{loss recovered} \;=\; \frac{\mathbb{E}[L_{\text{random}}] - \mathbb{E}[L_{\text{scrubbed}}]} {\mathbb{E}[L_{\text{random}}] - \mathbb{E}[L_{\text{model}}]} . \]

1 means the hypothesis explains all of the behaviour; 0 means it explains none of it. \(L_{\text{random}}\) is the loss when everything is resampled.

Toy example. A model computes \(y = 3x_1 + 0\cdot x_2\). The clean input is \(x=(1,4)\), so \(y=3\). A random other input is \(x'=(2,7)\). Hypothesis A: “only \(x_1\) matters”. Scrub \(x_2\) to 7: \(y = 3\cdot 1 + 0 \cdot 7 = 3\). Unchanged, so A passes. Hypothesis B: “only \(x_2\) matters”. Scrub \(x_1\) to 2: \(y = 6\). The output moved, so B fails. A real test resamples many \(x'\) and averages, because one lucky resample can hide a dependence.

From a neural network to the Atari

Components of the network	The candidate RAM cells of the game (for Pong: the two scores, paddle positions, ball position).
The behaviour	The value of output cells after the 15-frame horizon: cells that are driven by other cells but drive none themselves.
The hypothesis	A set of cells claimed to carry the behaviour. The TRUE one is the outputs plus all their ancestors in the exact data-flow graph.
Resampling from another input	Writing a different value into an out-of-circuit cell at \(f^*\): each value from the oracle set {0, base+17, base+37} except the current one.
Loss recovered	Scrubbing-preserved performance: the share of (trial, output) pairs whose output byte equals the clean run.
A wrong hypothesis	The true circuit with one demonstrably live ancestor swapped for an irrelevant cell, keeping the size the same.

How we apply it to the VCS

On a network you must guess the hypothesis. On the VCS we can compute the true one and then check that scrubbing rewards it and punishes a false one. Our runner causal_scrubbing.jl does this per game:

1. State. The shared gameplay checkpoint at \(f^*\) (90 seeded random-action frames) and the 15-frame action tail.
2. True data-flow graph. For each candidate cell \(i\), write 0, base+17 and base+37 in turn, replay the tail, and mark an edge \(i\to j\) if any value changes cell \(j\). This is the same exact graph the connectomics and ACDC pages are scored against.
3. Behaviour and hypotheses. The outputs are the sinks of that graph that have at least one ancestor. The true circuit is the outputs plus every cell with a path to them. The wrong circuit drops one ancestor that is confirmed live (changing it alone changes an output) and adds an outside cell instead.
4. Scrubbing. For each hypothesis, every cell outside the circuit is overwritten with a resample value, while circuit cells keep their clean value. The trial set walks through every resample value of every outside cell (the others held at their first resample value), plus one trial with all outside cells at their first resample value. After each trial the tail is replayed and the output cells are compared with the clean run.
5. Verdict. A hypothesis passes only if every output is preserved in every trial (score exactly 1). The run also reports whether the true circuit scores higher than the wrong one, and a graded similarity over all candidate cells and over the whole screen.

Where we differ from the original. The resample values are not activations from other real inputs. They come from the oracle's fixed value set, the same set used to build the true graph. The hypothesis is also only a set of cells: it makes no claim about what each cell computes, so there is no “consistent” resampling of in-circuit cells. Because the true circuit is built from the same interventions, its pass is close to guaranteed; it can still fail if several scrubbed cells interact. The informative half of the experiment is the wrong hypothesis: it must fail, and it shows that the test can tell a right story from a wrong one.

Read the code

XAI the interpretability method   substrate the bit-exact emulator it runs on (jutari, Julia).

• XAIcausal_scrubbing.jl:289 — The scrub: overwrite every out-of-circuit cell, keep circuit cells, replay the tail.
• XAIcausal_scrubbing.jl:276 — The resample distribution per cell: {0, base+17, base+37} minus the current value.
• XAIcausal_scrubbing.jl:221 — The exact data-flow graph the true hypothesis is derived from.
• XAIcausal_scrubbing.jl:245 — Transitive closure: every cell with a path to an output joins the true circuit.
• XAIcausal_scrubbing.jl:352 — Chooses the behaviour: sink cells of the true graph that have at least one ancestor.
• XAIcausal_scrubbing.jl:426 — Builds true and wrong hypotheses, runs every scrub trial, computes pass/fail and the triad.
• substrateshared_testbed_impl.jl:115 — The shared gameplay checkpoint, action tail and clean continuation.
• substratejutari_oracle.jl:176 — Writes one byte into emulated RAM: the physical act of scrubbing a cell.
• substrateStellaEnvironment.jl:259 — One bit-exact video frame; each scrub trial replays 15 of them.
• substrateoracle_intervene.jl:242 — The exact intervention oracle behind the shared state's cause-density gate.

Reading the result

The figure shows two preservation maps for Pong (output cell × scrub trial): one for the true hypothesis and one for the wrong one. Expect the true map fully preserved and the wrong map broken in some trials, and the same split game after game in the scores. That pattern means two things. The test is sensitive: it catches a story that wrongly calls a live cell irrelevant. And it is exact here, because each resample is a real intervention on a fully observable machine, not an approximation. What a pass does not tell you is how the circuit computes the behaviour. It confirms the wiring, not the algorithm.

Pitfalls

• A passing test needs a failing control. If the resampling is too weak (for example, the pool of other inputs is too similar), every hypothesis passes. Always scrub a wrong hypothesis too.
• Over-large hypotheses pass trivially. “Everything matters” never fails a scrub. Pair the score with a measure of how small the hypothesis is.
• Resample distribution matters. Chan et al. resample from real inputs; we use a fixed oracle value set. Different distributions can expose different dependences, and Miller et al. show such details can flip verdicts.
• Wiring is not meaning. A set-of-cells hypothesis that passes says where the computation happens, not what it is. Testing a semantic claim needs a richer hypothesis (see DAS).

Try it yourself

julia --project=jutari tools/xai_study/phaseC_mechanistic/causal_scrubbing.jl --games core

Writes causal_scrubbing_<game>.{json,npz} and causal_scrubbing_core_summary.json into tools/xai_study/phaseC_mechanistic/out/. --selftest runs a quick check on Pong.

Original papers & further reading

• Chan, Garriga-Alonso, Goldowsky-Dill, Greenblatt, Nitishinskaya, Radhakrishnan, Shlegeris & Thomas (2022). Causal Scrubbing: a method for rigorously testing interpretability hypotheses. Redwood Research / Alignment Forum. — www.alignmentforum.org/posts/JvZhhzycHu2Yd57RN/causal-scrubbing-a-method-for-rigorously-testing
• Geiger, Ibeling, Zur, Chaudhary, Chauhan, Huang, Arora, Wu, Goodman, Potts & Icard (2023). Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability. (follow-up: puts scrubbing, patching and DAS in one framework) — arxiv.org/abs/2301.04709
• Miller, Chughtai & Saunders (2024). Transformer Circuit Faithfulness Metrics are not Robust. (critique: ablation details change the verdict) — arxiv.org/abs/2407.08734

Read next: ACDC — automatic circuit discovery · Interchange interventions / DAS · Path patching / IOI circuit · Activation patching / causal mediation