cf. Jonas & Kording 2017
Connectomics tries to map the wiring of a system by watching how its parts affect each other. In neuroscience it means reconstructing which neurons connect to which. Here we do the same for the machine's memory. We change one RAM cell, re-run the program, and record which other cells change as a result. Each such dependency is drawn as a directed edge from cause to effect. The full set of edges is the program's data-flow graph, its 'connectome'. The idea is that the pattern of who-influences-whom should reveal the computation, without reading the code. This is a classic move in systems neuroscience: infer function from structure. On the VCS we have the true wiring from the disassembly, so we can check whether the recovered graph is right. The method is attractive because it needs no labels and no model of what the program means. It only needs the ability to perturb and observe. In practice it recovers a graph that looks plausible but contains many edges the real program does not have, because a single perturbation ripples through shared clocks and buffers.
Two adjacency matrices over the candidate RAM cells: the true data-flow graph and the graph this method recovered. A bright cell (row = cause, column = effect) is an edge; the difference between the two panels is the error. dataflow_graph_F1_vs_oracle = 0.000 — this example only (pong, state gameplay(seed=0,prefix=90)+15); the audit aggregate is below.
New to explainable AI? This block explains the method as it is used on neural networks, then how we run the very same method on an Atari 2600, where — unlike for a network — we know the right answer. Every implementation step links to the code. See also the XAI primer for the shared vocabulary.
A connectome is a wiring diagram: a list of which parts of a system talk to which other parts. Neuroscientists build it for brains. They draw one dot per neuron and one arrow per connection. The hope is simple: if we know who is wired to whom, we can read off how the system computes. One way to find the arrows is to look at the anatomy. Another is to perturb: nudge one unit, watch which other units react, and draw an arrow from the nudged unit to every unit that moved. The result is a directed graph of influence.
For a neural network you already have the anatomy: the weight matrices are the connectome, and they are fully known. Yet nobody claims that printing all the weights explains the network. That is the lesson of Jonas & Kording, who mapped a whole microprocessor and still could not say how it ran a game. On the Atari we ask a sharper question. If we recover the wiring by cheap perturbations, the way it is done in practice, how many of the arrows do we get right? Because the machine can be re-run exactly, we can count every right and every wrong arrow.
Neuroscience origin. The first complete connectome was that of the worm C. elegans: 302 neurons, about 7000 synapses, mapped by electron microscopy in the 1980s. Its behaviour is still not fully explained by that map. Perturbation-based mapping (stimulate one cell, record the others) gives the effective connectivity instead of the anatomical one.
Neural-network analogue. In a network the anatomical connectome is the set of non-zero weights. The perturbation version is what interpretability calls circuit discovery: change one activation \(h_i\), see which downstream activations \(h_j\) change. Write the intervention as \(\mathrm{do}(h_i := v)\) (“force \(h_i\) to the value \(v\), leave everything else to the network”). The recovered edge set is
\[ E_{\text{rec}} = \{\, i\to j \;:\; h_j^{(L)}\big(\mathrm{do}(h_i := h_i+\delta)\big) \neq h_j^{(L)} \,\}, \]where \(L\) is the latency (how many steps later you look) and \(\delta\) is the size of the nudge. Given a true edge set \(E_{\text{true}}\) you score it like any detector:
\[ P = \frac{|E_{\text{rec}}\cap E_{\text{true}}|}{|E_{\text{rec}}|},\quad R = \frac{|E_{\text{rec}}\cap E_{\text{true}}|}{|E_{\text{true}}|},\quad F_1 = \frac{2PR}{P+R}. \]Toy example. Three units in a chain, each copying the previous one with a one-step delay: \(a \to b \to c\). Nudge \(a\) and look one step later: only \(b\) moved. Nudge \(b\): only \(c\) moved. So \(E_{\text{rec}} = \{a\to b,\; b\to c\}\). Now look many steps later instead: nudging \(a\) moves \(b\) and \(c\). The long-horizon truth is \(E_{\text{true}} = \{a\to b,\; b\to c,\; a\to c\}\). Then \(P = 2/2 = 1\), \(R = 2/3\) and \(F_1 = 0.8\). Make it harder: let \(b\) copy \(a\) only when \(a > 100\). With \(a = 50\) and a nudge of \(+17\), the gate never opens and even \(a\to b\) is missed. A single small nudge with a short wait sees only part of the wiring.
| The neuron | One candidate RAM byte. The VCS has 128 bytes of working memory (RAM). Each game's candidate list (e.g. ball x, paddle y, score) comes from published RAM annotations (OCAtari / AtariARI). |
|---|---|
| The synapse | A data-flow edge \(i\to j\): the game program (the ROM, the read-only cartridge code) reads byte \(i\) and, directly or through other bytes, writes a value into byte \(j\). |
| The stimulation electrode | intervene_ram!: overwrite one RAM byte in the emulator at frame \(f^*\). A frame is one 1/60 s video picture, the unit of game time. |
| The recording | A full snapshot of RAM (and screen) after re-running the bit-exact emulator jutari from that frozen state. |
| The true connectome | The oracle graph: the same probe done exhaustively (three values, full 15-frame horizon). Each probe is a real re-run of the real ROM, so every edge in it is a true causal dependency. |
The runner A1_connectomics.jl does the whole experiment on one frozen game state.
That state is shared with every other method in the study:
Why the gap? The VCS program updates many bytes only on certain frames, behind comparisons and timers. A one-frame wait misses every influence that takes longer to arrive. A single nudge of \(+17\) misses every influence that only fires for other values. And an effect that shows up after one frame but is undone by frame 15 becomes an arrow the truth does not have. The method sees a small, partly wrong slice of a richly connected machine.
XAI the interpretability method substrate the bit-exact emulator it runs on (jutari, Julia).
A1_connectomics.jl:331 — The method under test: one do(i := base+17) per cell, one-frame latency, edge if another cell differs.A1_connectomics.jl:305 — The ground-truth graph: exhaustive do-set {0, +17, +37}, full horizon, edge if any value moves the cell.A1_connectomics.jl:243 — One probe: deep-copy the frozen machine, poke one RAM byte, step the tail, snapshot.A1_connectomics.jl:368 — Precision, recall, F1 and graph-edit distance over off-diagonal edges.A1_connectomics.jl:579 — The held-out base+37 graph used to test whether the recovery generalises.A1_connectomics.jl:489 — One game end to end: shared state, bit-exact check, both graphs, positive control, triad.shared_testbed_impl.jl:71 — The seeded random joystick stream (90 + 15 frames) every method shares.shared_testbed_impl.jl:115 — Builds the shared state and runs the oracle cause-density gate.jutari_oracle.jl:176 — The “electrode”: write one byte into the console's RAM (the A1 runner carries an identical local copy).StellaEnvironment.jl:259 — One video frame of the bit-exact emulated VCS: CPU, TIA video chip and RIOT timer.The figure shows the game frame and two square matrices over the same candidate bytes. Rows are the cause byte, columns the effect byte, and a bright square is an arrow. The left matrix is the oracle's true graph, the right one the recovered graph. The diagonal (a poked byte changing itself, unless the program overwrites it) is not scored. Read the error as the squares that are bright in one matrix but dark in the other. In Pong the true graph has whole bright rows, because the score bytes and the ball bytes reach many other cells within 15 frames. The one-frame recovery is empty off the diagonal. It misses the wiring rather than inventing it, so recall collapses and \(F_1\) lands at the bottom of the battery.
julia --project=jutari tools/xai_study/phaseA_kording/A1_connectomics.jl --games core
Writes A1_<game>.{json,npz} per game plus the combined A1_connectomics.{json,npz} into tools/xai_study/phaseA_kording/out/. Use --game pong for one game, or --selftest to run the checks without writing. Each game costs a few hundred short emulator re-runs (minutes on one core).
Read next: A2 · Single-unit lesions · A6 · Granger causality · A4 · Pairwise correlations · ACDC — automatic circuit discovery
The score is a data-flow-graph F1 against the true read/write graph from the disassembly. An edge means that perturbing one RAM cell changes another. We build the recovered graph by single-shot perturbation and compare its edges to the true edges, taking the harmonic mean of precision, the fraction of recovered edges that are real, and recall, the fraction of real edges recovered. Faithfulness for this method is that edge F1: a graph-valued explanation is graded as a graph, not as a heat-map. The comparison is always against the intervention oracle's true wiring, never against another method. The hard part is that a single perturbation ripples through shared clocks and buffers, so the recovered graph contains many edges the real program does not have. Precision suffers, and the F1 stays low. There is no separate position hard-case here, because the method never claims a sprite position; it claims a wiring diagram. The audit box below reports the actual F1, aggregated across the scored games, so no single game dominates. Read the recovered-versus-true graph as the error: every edge in one panel but not the other is a mistake the metric counts.
The score is measured against the §1 intervention oracle — never against another interpretability method. F (faithful) is always vs the oracle; see the execution stack. How each of F / S / M is actually computed for this method (and whether it matches the paper) is in the box just below; the numbers are in the In the audit box under it.
The exact formula this method uses for each score, read from its runner, and
whether it matches the paper's §3 (F ∧ S ∧ M triad) definition.
From the committed audit
fsm_math_phaseA.json.
| Axis | Formula (as computed) | What it measures | Matches §3? |
|---|---|---|---|
| F faithfulness | \(F = F_1^{\mathrm{edge}}(\hat{E}, E^{\star})\) | F1 of the single-shot recovered data-flow edges against the exact-oracle true data-flow graph over 8 candidate cells. | ✓ matches Graph-valued primitive = edge F1 vs true data-flow graph, exactly the paper's connectomics case. Runner A1_connectomics.jl,; record out/A1_pong.json extra.triad.F=0.0 / graph.f1. how it's measured — call stack:
|
| S sufficiency | \(S = \dfrac{\#\{(i,j):\ \hat{E}_{ij}=E^{\star,\mathrm{held}}_{ij}\}}{\#\,\mathrm{off\text{-}diag}} \in [0,1]\) | Off-diagonal edge agreement between the base+17 recovered graph and the held-out base+37 true edge set (unseen intervention value). | ✓ matches Held-out edge agreement between the base+17 recovered graph and the held-out base+37 true edge set: a non-null held-out sufficiency. §3 (sec:triad) admits the fraction-within-tolerance [0,1] special case of the held-out predictive test, which this off-diagonal edge-agreement fraction satisfies. A1_connectomics.jl,; record S=0.839. how it's measured — call stack:
|
| M minimality | \(M = |U^{\star}| / |\hat{U}| = |E^{\star}| / |\hat{E}| \in (0,1]\) | True-graph edge count over the number of recovered edges (edge-level |U*|/|U_hat|). | ✓ matches Runner now emits M = |U*|/|U_hat| over edges (M_note '|true edges|=12 / |discovered edges|=0'); the paper's minimality ratio applied at the edge level. Record out/A1_pong.json extra.triad.M (null in pong when 0 edges discovered). Replaced the former 1 − FP-rate (edge precision). how it's measured — call stack:
|
✓ matches = the same quantity as §3; ◐ partial = the same kind of estimator but a differing detail; ✗ does not match = a different quantity (see the note). Definitions: F = agreement with the oracle's true causal effects Δy(u); S = held-out predictive score in [−1, 1]; M = |U*|/|Û| (true-minimal-set size / named-set size).
This is the method's entry in the actual cross-method audit — scored on the paper's correctness triad, each axis a mean over 38 of the 44 scored games (38 committed §R records), not the single example shown above. Tradition: intervention. The example figure (Pong) is one of those records.
F faithfulness (scored vs the oracle for every method) · S sufficiency (held-out predictive score in [−1, 1]; a negative value means the explanation predicts held-out interventions worse than the unperturbed baseline; reported for the predictive methods across all three phases where the calibration/held-out split is defined) · M minimality (true-minimal-set / named-set; where the method names a cause set) — n/a otherwise, per the paper's F ∧ S ∧ M triad.
Source: leaderboard.json · the whole leaderboard is on the
methods page and the
Paper 2 audit.
This method's faithfulness on each of the 38 scored games (all-regime F; this method has no content-vs-position split (see the note below)). Click a header to
sort. Every number is read from site_data.json
(methods.A1_connectomics.per_game).
content-F and position-F are n/a for this method. The content-vs-position split applies only to pixel-attribution methods — those that produce an attribution map over the framebuffer, whose score can be read separately on the smooth content outputs and the discrete position outputs (where the naive gradient is provably zero). This method does not attribute to a pixel output; it is graded by its own fidelity metric (dataflow_graph_F1_vs_oracle), reported as the F column, so there is no content or position regime to separate.
M is n/a on 15 of these games. Minimality is M = |U*|/|Û|, the size of the true minimal cause set over the size of the set the method actually names. On a game where the method names nothing (|Û| = 0 — it discovers no circuit, its attribution map is all-zero, or it decodes no cell), that ratio is undefined, so M is left blank rather than scored. The aggregate M is the mean over the 23 games where the method did name a cause set.
| Game | F (all) | content-F | position-F | S | M | Note |
|---|---|---|---|---|---|---|
| Q*bert | 1.000 | n/a | n/a | 1.000 | 1.000 | recovers the true causes |
| Bowling | 0.667 | n/a | n/a | 1.000 | 1.000 | |
| Video Pinball | 0.667 | n/a | n/a | 0.976 | 1.000 | |
| Tennis | 0.533 | n/a | n/a | 0.967 | 1.000 | |
| Space Invaders | 0.500 | n/a | n/a | 0.982 | 1.000 | |
| Chopper Command | 0.452 | n/a | n/a | 0.990 | 1.000 | |
| Road Runner | 0.444 | n/a | n/a | 0.987 | 1.000 | |
| Hero | 0.400 | n/a | n/a | 0.989 | 1.000 | |
| Private Eye | 0.364 | n/a | n/a | 1.000 | 1.000 | |
| Double Dunk | 0.333 | n/a | n/a | 0.962 | 1.000 | |
| Krull | 0.333 | n/a | n/a | 0.975 | 0.333 | |
| Riverraid | 0.320 | n/a | n/a | 0.972 | 1.000 | |
| Pitfall | 0.308 | n/a | n/a | 0.975 | 1.000 | |
| Breakout | 0.286 | n/a | n/a | 1.000 | 1.000 | |
| Seaquest | 0.250 | n/a | n/a | 0.984 | 1.000 | |
| Air Raid | 0.222 | n/a | n/a | 0.955 | 1.000 | |
| Asterix | 0.222 | n/a | n/a | 0.972 | 1.000 | |
| Carnival | 0.200 | n/a | n/a | 0.997 | 1.000 | |
| Yars' Revenge | 0.200 | n/a | n/a | 0.949 | 1.000 | |
| Montezuma's Revenge | 0.125 | n/a | n/a | 0.976 | 1.000 | |
| Boxing | 0.111 | n/a | n/a | 0.891 | 1.000 | |
| Demon Attack | 0.105 | n/a | n/a | 0.962 | 1.000 | |
| Berzerk | 0.043 | n/a | n/a | 0.957 | 1.000 | no true-cause signal |
| Alien | 0.000 | n/a | n/a | 1.000 | n/a | no true-cause signal |
| Assault | 0.000 | n/a | n/a | 0.990 | n/a | no true-cause signal |
| Atlantis | 0.000 | n/a | n/a | 1.000 | n/a | no true-cause signal |
| Centipede | 0.000 | n/a | n/a | 0.972 | n/a | no true-cause signal |
| Enduro | 0.000 | n/a | n/a | 0.500 | n/a | no true-cause signal |
| Frostbite | 0.000 | n/a | n/a | 0.977 | n/a | no true-cause signal |
| Ice Hockey | 0.000 | n/a | n/a | 0.922 | n/a | no true-cause signal |
| Jamesbond | 0.000 | n/a | n/a | 0.981 | n/a | no true-cause signal |
| Kangaroo | 0.000 | n/a | n/a | 0.995 | n/a | no true-cause signal |
| Ms. Pac-Man | 0.000 | n/a | n/a | 0.947 | n/a | no true-cause signal |
| Pacman | 0.000 | n/a | n/a | 0.982 | n/a | no true-cause signal |
| Phoenix | 0.000 | n/a | n/a | 0.992 | n/a | no true-cause signal |
| Pong | 0.000 | n/a | n/a | 0.839 | n/a | no true-cause signal |
| Skiing | 0.000 | n/a | n/a | 0.917 | n/a | no true-cause signal |
| Venture | 0.000 | n/a | n/a | 0.938 | n/a | no true-cause signal |
The path from this method's runner (run_game) into the bit-exact VCS substrate, and the
computation it involves. Every step links to the exact source on main. From
callstack_phaseA.json.
build_a1_shared_state(game) -> build_shared_testbed(game; ...) runnerfunction build_shared_testbed(game; ...) runnercause_density = count(>(floor), deltas) runnercheckpoint = boot_replay(spec, actions, target_frame) runnerenv_reset!(env; boot_noop_steps=60, boot_reset_steps=4) runnerenv_step!(env, Int(actions[i])) jutariget_ram(env) jutariat_target = continue_from(checkpoint, Int[]) runnerrecovered_dataflow_graph(checkpoint, tail, cands, at_target; delta=17, latency=1) runnertrue_dataflow_graph(checkpoint, tail, cands, at_target) runnerintervene_continue(checkpoint, c.ram_index, v, tail) runnerscore_graph(rec_graph, true_graph) runnerScored against the exact intervention oracle Δy(u): run_intervention(checkpoint, actions, target_frame, horizon, cause).
The figure is generated from the committed record by
docs/gen_method_figures.py; the game frame
and each RAM cell's screen footprint are produced by
render_scenes.jl /
cell_footprints.jl.