ACDC — automatic circuit discovery

Mechanistic family · circuit discovery, automated

Learn the method — a beginner's guide

New to explainable AI? This block explains the method as it is used on neural networks, then how we run the very same method on an Atari 2600, where — unlike for a network — we know the right answer. Every implementation step links to the code. See also the XAI primer for the shared vocabulary.

The idea in plain words

A trained network is a huge web of computations. Mechanistic interpretability hopes that a specific behaviour, such as “complete the name in this sentence”, is carried by a small sub-web of that computation. That sub-web is called a circuit. For years, people found circuits by hand: guess a component, knock it out, look at the output, repeat. That is slow and depends on the researcher's intuition.

ACDC automates the search. Draw the whole model as a graph: nodes are components (attention heads, MLP blocks), and an edge means “this node's output is an input to that node”. Then walk over the edges and cut each one in turn. If cutting an edge barely changes the model's output, the edge is not needed and is pruned for good. If it changes the output by more than a threshold \(\tau\), it stays. What survives is the discovered circuit. Turning the dial \(\tau\) gives a whole family of circuits, from tiny (large \(\tau\)) to almost the full model (small \(\tau\)).

How it works on a neural network

“Cutting” an edge does not mean setting it to zero. Zero is often an input the model has never seen. ACDC instead uses resample ablation: the signal on the edge is replaced by the value it takes on a corrupted input (for example, the same sentence with a different name). Let \(G\) be the full model, \(H\) the current candidate circuit, and \(H \setminus e\) the circuit with edge \(e\) cut. ACDC measures how far the output distribution moves, with the Kullback–Leibler divergence \(D_{\mathrm{KL}}\) (a standard distance between two probability distributions), and prunes \(e\) if

\[ D_{\mathrm{KL}}\big(G \,\|\, H\setminus e\big) \;-\; D_{\mathrm{KL}}\big(G \,\|\, H\big) \;<\; \tau . \]

Edges are visited from the output backwards, and every prune is kept, so later decisions are made on an already-thinned graph.

Toy example. A network has input \(x\), two hidden units \(A\) and \(B\), and output \(y\). The four edges are \(x\!\to\!A\), \(x\!\to\!B\), \(A\!\to\!y\), \(B\!\to\!y\). Resample-ablating each one moves the output by 1.4 (for \(A\!\to\!y\)), 0.02 (\(B\!\to\!y\)), 1.0 (\(x\!\to\!A\)) and 0.5 (\(x\!\to\!B\)). With \(\tau = 0.1\), \(B\!\to\!y\) is pruned. Now \(B\) feeds nothing, so cutting \(x\!\to\!B\) changes nothing either, and it is pruned too. The circuit is \(x \to A \to y\). Note how the second decision depended on the first: that is the iterative part.

From a neural network to the Atari

The model's computational graph	The data flow between candidate RAM cells of the cartridge program. For Pong these are the 8 cells the labelling step named: the two scores, the paddle and ball positions.
A node (attention head, MLP)	One candidate RAM byte, e.g. RAM[49] = ball x.
An edge \(i \to j\)	“The value of cell \(i\) at frame \(f^*\) influences the value of cell \(j\) 15 frames later”. This is a total effect over the horizon, not one direct wire.
Corrupted input / resample ablation	Overwrite cell \(i\) at \(f^*\) with a corrupted value (from a donor run if it differs, else the clean value + 17), then re-run the real ROM.
Output metric (KL divergence)	The absolute change of child cell \(j\)'s byte after the horizon: \(|y_j(\text{ablated}) - y_j(\text{clean})|\).
The true circuit (unknown for a network)	The exhaustive data-flow graph: edge \(i\to j\) exists iff setting cell \(i\) to 0, base+17 or base+37 changes cell \(j\) on the bit-exact re-run.

How we apply it to the VCS

On a language model nobody knows the “real” circuit, so ACDC can only be judged against other human guesses. On the VCS the program's data flow can be measured exactly, because jutari re-runs the ROM bit-for-bit. Our runner acdc.jl does the following:

1. State. It boots the game, plays the shared seeded random-action stream for 90 frames, and checkpoints the machine at \(f^*\). The remaining 15 frames of the stream are the tail. It asserts that two fresh boots give identical RAM and screen before scoring anything.
2. Edge effects. For every candidate cell \(i\), it writes one corrupted value into cell \(i\), plays the tail once, and reads every candidate cell \(j\). That fills a whole row of the effect matrix \(E[i,j]\) with one re-run. The corrupted value comes from a donor run that switches to the RIGHT action at \(f^*\); because that donor is identical to the clean run up to \(f^*\), the synthetic fallback base+17 is what is normally used. The record counts how often the donor really diverged.
3. Threshold sweep. The discovered circuit at \(\tau\) is every off-diagonal pair with \(E[i,j] > \tau\). Thresholds are placed between consecutive distinct effect sizes, so the sweep walks from “every edge with any effect” down to “nothing”.
4. Scoring. Each circuit is compared with the exhaustive true graph: precision, recall, F1, plus an ROC curve and its area. The headline is the best F1 over the sweep.
5. Sufficiency. At the best \(\tau\), every source cell with no surviving outgoing edge is resample-ablated at the same time, the tail is replayed, and we count the fraction of candidate cells that still match the clean run (“scrubbing-preserved performance”).
6. Positive control. Running the same keep-rule with the full oracle value set and \(\tau=0\) reproduces the true graph exactly. This proves the scorer can reward a perfect discovery.

Where we differ from the original. Our ACDC is single-shot: one ablation per source, then pure thresholding. It does not prune iteratively, so it skips the “dangling node” step of the toy example. The metric is a byte difference, not a KL divergence. And an edge is a total effect over 15 frames, because RAM cells interact through many CPU instructions, not through one weight. These choices make it cheap and comparable with the oracle. They also explain a typical failure: a single corrupted value can miss edges that only fire for other values, which costs recall.

Read the code

XAI the interpretability method   substrate the bit-exact emulator it runs on (jutari, Julia).

• XAIacdc.jl:417 — The method: one resample-ablation and real-ROM re-run per source cell fills a row of \(E[i,j]\).
• XAIacdc.jl:404 — Chooses the corrupted value: the donor's byte if it differs, else base+17.
• XAIacdc.jl:442 — The keep rule: edge \(i\to j\) survives iff its effect exceeds \(\tau\).
• XAIacdc.jl:482 — The threshold schedule: one \(\tau\) between each pair of distinct effect sizes.
• XAIacdc.jl:362 — The ground-truth circuit: exhaustive interventions {0, base+17, base+37} on each cell.
• XAIacdc.jl:546 — Sufficiency check: scrub all non-circuit sources at once and count preserved cells.
• XAIacdc.jl:647 — One game end to end: bit-exact check, checkpoint, truth, ACDC, sweep, positive control, triad.
• substrateshared_testbed_impl.jl:115 — The shared gameplay state every method sits on (seeded actions, \(f^*\)=90, 15-frame horizon).
• substrateStellaEnvironment.jl:259 — One video frame of the bit-exact machine; every re-run in ACDC is a loop of these.
• substrateoracle_intervene.jl:242 — The exact intervention oracle used to build the shared state and its cause-density gate.

Reading the result

The figure shows the threshold sweep: precision and recall of the discovered edges against the true data-flow graph as \(\tau\) falls. A perfect method would hit the top-right corner at some \(\tau\); the positive control does exactly that. Expect ACDC's curve to sit lower. Precision is usually decent, because an edge with a visible effect under one corruption is usually real. Recall is the weak side: an edge that only shows up for a different value, or that the program re-derives from another cell, looks dead under a single resample and gets pruned. Read the scrubbing-preserved number beside it: a circuit can score a good F1 and still fail to reproduce the behaviour on its own.

Pitfalls

• One corruption is not all corruptions. An edge whose effect depends on the value chosen can be invisible under the one resample used. The exhaustive oracle tries several values; ACDC tries one.
• The threshold is a free parameter. There is no correct \(\tau\). Reporting only the best-F1 point uses the ground truth to pick it, which a real network user cannot do.
• Total vs direct effects. On the VCS an edge means “influences after 15 frames”. Indirect paths (\(i\to k\to j\)) appear as edges \(i\to j\) too, in both the discovered and the true graph.
• Minimal is not sufficient. A small, clean-looking circuit can still fail when everything outside it is scrubbed. Always check sufficiency separately (see causal scrubbing).
• Metric choice changes the circuit. Miller et al. show that small changes in how ablations and metrics are defined change which circuit is found. Our byte-difference metric is one choice among many.

Try it yourself

julia --project=jutari tools/xai_study/phaseC_mechanistic/acdc.jl --games core

Writes acdc_<game>.{json,npz} and acdc_core_summary.json into tools/xai_study/phaseC_mechanistic/out/. Add --selftest to run the positive control on the pilot game without writing anything.

Original papers & further reading

• Conmy, Mavor-Parker, Lynch, Heimersheim & Garriga-Alonso (2023). Towards Automated Circuit Discovery for Mechanistic Interpretability. NeurIPS. — arxiv.org/abs/2304.14997
• Syed, Rager & Conmy (2023). Attribution Patching Outperforms Automated Circuit Discovery. — arxiv.org/abs/2310.10348
• Miller, Chughtai & Saunders (2024). Transformer Circuit Faithfulness Metrics are not Robust. (critique of how circuits are scored) — arxiv.org/abs/2407.08734

Read next: Causal scrubbing · Path patching / IOI circuit · Activation patching / causal mediation · A1 · Connectomics / data-flow graph