cf. Jonas & Kording 2017
This method studies the correlation structure across the machine's cells. In neuroscience the 'spike-word' or pairwise-correlation analysis looks at which units fire together, and reads coordinated firing as evidence of shared function or coupling. We compute the same statistics over RAM. For every pair of cells we measure how their values co-vary across the trajectory, and we also measure the global, population-level correlation. The familiar finding in neural data is weak pairwise but strong global correlation, and the VCS reproduces that signature. The idea is that cells which move together are functionally linked, so the correlation matrix should approximate the true coupling. The method is purely observational: it never intervenes, it only watches. That is its limit. On a clock-locked machine almost everything co-varies with almost everything else, driven by shared timing signals rather than by direct influence. So the correlation structure is real but mostly reflects common drivers, not causal links. It detects that the machine is active and coordinated, but it does not separate a true dependency from two cells that merely share a clock.
Two adjacency matrices over the candidate RAM cells: the true data-flow graph and the graph this method recovered. A bright cell (row = cause, column = effect) is an edge; the difference between the two panels is the error. A4_faithfulness_spearman_corr_vs_oracle_coupling = 0.633 — this example only (pong, state gameplay(seed=0,prefix=90)+15); the audit aggregate is below.
New to explainable AI? This block explains the method as it is used on neural networks, then how we run the very same method on an Atari 2600, where — unlike for a network — we know the right answer. Every implementation step links to the code. See also the XAI primer for the shared vocabulary.
Record many neurons at once and you get a table: one row per moment in time, one column per neuron. A natural first question is which neurons go up and down together? Measure that for every pair and you get a correlation matrix. Groups of units that move together are candidates for working together. In neuroscience the patterns of joint firing are called spike words, and their statistics have been studied in detail.
A famous finding is “weak pairwise, strong global”. Any two neurons are only weakly correlated, yet the whole population sits in a few coordinated states far more often than chance. The tempting reading is that the correlation structure reveals the coupling, the true influence of one unit on another. But correlation is a passive measurement. Two units can move together because one drives the other, or because both are driven by a third thing. On the VCS almost everything is driven by the same clock, and the oracle can tell the two cases apart.
Neuroscience origin. Schneidman et al. recorded about 40 retinal neurons and found pairwise correlations that were small, yet a model built only from them predicted the frequent population-wide patterns. Tkačik et al. extended this to larger populations. Jonas & Kording ran the same statistics on transistors of a 6502 chip and found the same “weak pairwise, strong global” picture, in a system whose real wiring is known and looks nothing like a brain.
Neural-network analogue. Correlating unit activations over a dataset is common: to find redundant units, to match units across two trained networks (Li et al.), or to compare whole layers (SVCCA, CKA). For units \(i,j\) with activations \(x_i(t)\) over inputs \(t\):
\[ C_{ij} = \rho\big(x_i, x_j\big), \qquad \mathrm{PR} = \frac{\big(\sum_k \lambda_k\big)^2}{\sum_k \lambda_k^2}, \]where \(\lambda_k\) are the eigenvalues of \(C\). The participation ratio PR counts the effective number of independent directions. A PR far below the number of units means a few global modes dominate (the same idea as keeping the top principal components).
Toy example 1 (common driver). Let a clock run \(t = 1,\dots,5\). Unit \(x = t\) and unit \(y = 2t + 1\). Neither reads the other. Still \(\rho(x,y) = 1\). If you overwrite \(x\), \(y\) does not change: the true coupling is \(0\). A correlation of 1, a coupling of 0.
Toy example 2 (weak pairwise, strong global). Three units, every pair correlated \(0.3\). The eigenvalues of \(C\) are \(1.6, 0.7, 0.7\), so \(\mathrm{PR} = 3^2 / (1.6^2 + 0.7^2 + 0.7^2) = 9 / 3.54 \approx 2.5\). With 300 such units, the top eigenvalue is \(1 + 299 \cdot 0.3 \approx 91\), the other 299 are \(0.7\), and PR is about 11, far below 300: many weak pairs add up to one strong collective mode.
| The neuron | One candidate RAM byte of the VCS's 128-byte working memory, named by published annotations (OCAtari / AtariARI); for the population statistics, every RAM byte that varies. |
|---|---|
| The multi-electrode recording | A RAM tape: all 128 bytes logged after every frame (one 1/60 s picture) while the bit-exact emulator plays the shared input stream, 150 frames for most games. |
| The spike word | The joint pattern of byte values at one frame; the pairwise statistic is the Pearson correlation of two bytes' time series. |
| The true coupling | The oracle: overwrite byte \(i\) at frame \(f^*\) with \(0\) or base\(+17\), re-run the real ROM 15 frames, measure \(|\Delta|\) of byte \(j\). Symmetrised as \(\max(i\to j,\ j\to i)\). |
| Known cell types | Concept families from the labels: all enemy_* bytes form one true group, all player_* bytes another, and so on. |
The runner A4_correlations.jl has an observational half (the method) and an
interventional half (the truth). Both use the shared game state: boot the cartridge (the ROM), play
90 frames of seeded random joystick input to reach \(f^*\), keep the next 15 frames as the horizon,
and confirm that two fresh replays agree byte for byte.
a4_record_ram steps the emulator along the same input stream
(no-ops after the gameplay frames) and stores the 128 RAM bytes after every frame. The length is
set per game (150 frames for most, longer for Q*bert and Ms. Pac-Man).pairwise_corr computes the Pearson matrix over the
candidate bytes. A byte that never changes has zero variance and is given correlation 0 with
everything.How this differs from the neural original. Our units are bytes with up to 256 values, not binary spikes, so we use Pearson correlation rather than a maximum-entropy model of spike words. And the “true coupling” is a measured total effect over 15 frames, which includes indirect paths, not a list of direct synapses.
XAI the interpretability method substrate the bit-exact emulator it runs on (jutari, Julia).
A4_correlations.jl:241 — The recording: a (frames × 128) RAM tape from the bit-exact emulator.A4_correlations.jl:391 — The method: Pearson correlation matrix over the candidate bytes.A4_correlations.jl:447 — Clusters vs concept families: Rand index, purity, over-grouping ratio.A4_correlations.jl:531 — Weak-pairwise / strong-global: eigenvalue shares and participation ratio, candidates and full population.A4_correlations.jl:579 — The ground truth: do(0) / do(+17) per byte, 15-frame re-run, |Δ| of every other byte; +37 held out.A4_correlations.jl:650 — F = Spearman(|corr|, coupling) over pairs; S on the held-out value; M named vs true pairs.shared_testbed_impl.jl:115 — The shared seeded-gameplay state and its oracle cause-density gate.triad_sm.jl:57 — The shared minimality score M = |U*| / |Û| used by every method.StellaEnvironment.jl:259 — One video frame of the bit-exact emulated VCS (called once per recorded frame).StellaEnvironment.jl:350 — Read the 128 RAM bytes: one row of the recording.The figure shows two square matrices over the same candidate bytes. Left: the oracle's true coupling; a bright square means overwriting one byte changes the other within 15 frames. Right: the measured correlation. The diagonal of the right matrix is always 1 and is not scored. If the two matrices lit up the same off-diagonal squares, correlation would be a faithful map of coupling. In Pong most candidate bytes are zero in both, and both light up the same small group of ball and paddle bytes, so the ranking is partly right. But inside that group the two disagree: correlation gives every pair a similar, high value, because the bytes all move with the same ball, while the oracle says only one pair is strongly coupled, two weakly, and the player paddle not at all. Correlation found the group, not the wiring inside it. That is the partial score in the audit box.
julia --project=jutari tools/xai_study/phaseA_kording/A4_correlations.jl --games core
Writes A4_<game>.{json,npz} per game into tools/xai_study/phaseA_kording/out/. --tau-corr 0.7 sets the clustering threshold, --traj-frames N the recording length, --selftest runs the checks on the first game without writing.
Read next: A6 · Granger causality · A7 · Dimensionality reduction (NMF/PCA) · A3 · Tuning curves · A1 · Connectomics / data-flow graph
The score is a rank correlation between the measured cell-to-cell correlation structure and the oracle's true coupling. We compute how each pair of cells co-varies across the trajectory, assemble that into a coupling matrix, and compare it, by Spearman correlation, to the true coupling the oracle certifies. Faithfulness for this method is that correlation. It is only partial, because the method never intervenes; it only watches. On a clock-locked machine two cells can be strongly correlated because they share a driver, not because one influences the other. So the correlation matrix conflates true coupling with common drivers, and the score reflects that confusion. Sufficiency, whether the recovered structure can predict held-out interventions, is near zero for this method, because a correlation matrix carries no mechanism to re-generate behaviour. The grading is always against the oracle, not against another method. There is no sprite-position hard-case here; the output is a coupling matrix. The audit box reports the measured faithfulness and, where the paper defines it, the sufficiency, each aggregated across the scored games so no single game dominates the number.
The score is measured against the §1 intervention oracle — never against another interpretability method. F (faithful) is always vs the oracle; see the execution stack. How each of F / S / M is actually computed for this method (and whether it matches the paper) is in the box just below; the numbers are in the In the audit box under it.
The exact formula this method uses for each score, read from its runner, and
whether it matches the paper's §3 (F ∧ S ∧ M triad) definition.
From the committed audit
fsm_math_phaseA.json.
| Axis | Formula (as computed) | What it measures | Matches §3? |
|---|---|---|---|
| F faithfulness | \(F = \rho_{\mathrm{Spearman}}\!\left(|\mathrm{corr}|_{\mathrm{pairs}},\ \mathrm{coupling}_{\mathrm{pairs}}\right)\) | Spearman correlation between absolute pairwise data correlations and the oracle intervention-based cell-cell coupling over candidate-cell pairs. | ✓ matches Oracle 'coupling' is a bit-exact intervention effect (not a data correlation); F correlates the method's |corr| map against it. Correlation primitive on pairs. Runner A4_correlations.jl,; record out/A4_pong.json triad.F=0.633. how it's measured — call stack:
|
| S sufficiency | \(S = \rho_{\mathrm{Pearson}}\!\left(|\mathrm{corr}|_{\mathrm{pairs}},\ \mathrm{coupling}^{\mathrm{held\text{-}out}}_{\mathrm{pairs}}\right) \in [-1,1]\) | Pearson correlation of the pairwise-correlation map vs the held-out do(base+37) coupling. | ✓ matches Held-out predictive correlation in [-1,1] of the pairwise-correlation map vs the held-out do(base+37) coupling: the §3 (sec:triad) form (can be negative). A4_correlations.jl; record S=-0.170. how it's measured — call stack:
|
| M minimality | \(M = |U^{\star}| / |\hat{U}| \in (0,1]\quad(\text{over candidate-cell pairs})\) | Number of oracle-coupled cell pairs over the number of high-correlation pairs the method names. | ✓ matches Runner now emits M = |U*|/|U_hat| over pairs (M_note '|U*|=4 (oracle movers) / |U_hat|=6'), the paper's set-size ratio at the pair level. Record out/A4_pong.json extra.triad.M=0.667. Replaced the former 1 − correlation-overgrouping rate. how it's measured — call stack:
|
✓ matches = the same quantity as §3; ◐ partial = the same kind of estimator but a differing detail; ✗ does not match = a different quantity (see the note). Definitions: F = agreement with the oracle's true causal effects Δy(u); S = held-out predictive score in [−1, 1]; M = |U*|/|Û| (true-minimal-set size / named-set size).
This is the method's entry in the actual cross-method audit — scored on the paper's correctness triad, each axis a mean over 36 of the 44 scored games (36 committed §R records), not the single example shown above. Tradition: correlational. The example figure (Pong) is one of those records.
F faithfulness (scored vs the oracle for every method) · S sufficiency (held-out predictive score in [−1, 1]; a negative value means the explanation predicts held-out interventions worse than the unperturbed baseline; reported for the predictive methods across all three phases where the calibration/held-out split is defined) · M minimality (true-minimal-set / named-set; where the method names a cause set) — n/a otherwise, per the paper's F ∧ S ∧ M triad.
Source: leaderboard.json · the whole leaderboard is on the
methods page and the
Paper 2 audit.
This method's faithfulness on each of the 36 scored games (all-regime F; this method has no content-vs-position split (see the note below)). Click a header to
sort. Every number is read from site_data.json
(methods.A4_correlations.per_game).
content-F and position-F are n/a for this method. The content-vs-position split applies only to pixel-attribution methods — those that produce an attribution map over the framebuffer, whose score can be read separately on the smooth content outputs and the discrete position outputs (where the naive gradient is provably zero). This method does not attribute to a pixel output; it is graded by its own fidelity metric (A4_faithfulness_spearman_corr_vs_oracle_coupling), reported as the F column, so there is no content or position regime to separate.
M is n/a on 1 of these games. Minimality is M = |U*|/|Û|, the size of the true minimal cause set over the size of the set the method actually names. On a game where the method names nothing (|Û| = 0 — it discovers no circuit, its attribution map is all-zero, or it decodes no cell), that ratio is undefined, so M is left blank rather than scored. The aggregate M is the mean over the 35 games where the method did name a cause set.
S is negative on 8 of these games. Sufficiency asks whether the explanation can predict the outcome of held-out interventions it was not fit on; for this method S is the held-out predictive correlation between its claim and the true effect, so it lies in [−1, 1]. A value near 0 means the explanation predicts no better than the unperturbed output; a negative value means it predicts worse — its held-out claim is anti-correlated with the truth (named yet causally unused). The aggregate S is the mean over the scored games.
| Game | F (all) | content-F | position-F | S | M | Note |
|---|---|---|---|---|---|---|
| Pong | 0.633 | n/a | n/a | -0.170 | 0.667 | |
| Bowling | 0.609 | n/a | n/a | 0.527 | 0.667 | |
| Atlantis | 0.503 | n/a | n/a | 0.000 | 0.200 | |
| Enduro | 0.500 | n/a | n/a | -0.064 | 0.667 | |
| Boxing | 0.469 | n/a | n/a | 0.144 | 0.464 | |
| Video Pinball | 0.463 | n/a | n/a | 0.544 | 0.167 | |
| Air Raid | 0.445 | n/a | n/a | 0.398 | 0.250 | |
| Chopper Command | 0.416 | n/a | n/a | 0.179 | 0.256 | |
| Yars' Revenge | 0.401 | n/a | n/a | 0.250 | 0.286 | |
| Frostbite | 0.362 | n/a | n/a | -0.039 | 0.821 | |
| Venture | 0.354 | n/a | n/a | 0.141 | 0.396 | |
| Pitfall | 0.345 | n/a | n/a | 0.335 | 0.141 | |
| Seaquest | 0.304 | n/a | n/a | 0.033 | 0.429 | |
| Double Dunk | 0.295 | n/a | n/a | 0.081 | 0.111 | |
| Montezuma's Revenge | 0.280 | n/a | n/a | 0.322 | 0.867 | |
| Berzerk | 0.280 | n/a | n/a | 0.237 | 0.317 | |
| Demon Attack | 0.280 | n/a | n/a | 0.008 | 0.197 | |
| Jamesbond | 0.272 | n/a | n/a | 0.345 | 0.107 | |
| Carnival | 0.254 | n/a | n/a | 0.244 | 0.178 | |
| Ice Hockey | 0.230 | n/a | n/a | 0.329 | 0.321 | |
| Phoenix | 0.221 | n/a | n/a | 0.300 | 0.036 | |
| Breakout | 0.219 | n/a | n/a | -0.046 | 1.000 | |
| Riverraid | 0.136 | n/a | n/a | -0.027 | 0.110 | |
| Centipede | 0.129 | n/a | n/a | 0.143 | 0.143 | |
| Krull | 0.124 | n/a | n/a | 0.005 | 0.500 | |
| Space Invaders | 0.117 | n/a | n/a | 0.045 | 0.400 | |
| Asterix | 0.109 | n/a | n/a | 0.137 | 0.700 | |
| Assault | 0.107 | n/a | n/a | 0.247 | 0.222 | |
| Road Runner | 0.082 | n/a | n/a | -0.057 | 0.077 | |
| Q*bert | 0.049 | n/a | n/a | 0.081 | 0.009 | no true-cause signal |
| Tennis | 0.023 | n/a | n/a | 0.100 | 0.393 | no true-cause signal |
| Private Eye | 0.011 | n/a | n/a | 0.120 | 0.321 | no true-cause signal |
| Alien | 0.000 | n/a | n/a | 0.000 | 1.000 | no true-cause signal |
| Hero | 0.000 | n/a | n/a | -0.008 | 1.000 | no true-cause signal |
| Kangaroo | 0.000 | n/a | n/a | 0.000 | n/a | no true-cause signal |
| Skiing | 0.000 | n/a | n/a | -0.133 | 0.333 | no true-cause signal |
The path from this method's runner (compute_game) into the bit-exact VCS substrate, and the
computation it involves. Every step links to the exact source on main. From
callstack_phaseA.json.
build_a4_shared_state(game) -> build_shared_testbed(game; ...) runnerfunction build_shared_testbed(game; ...) runnercause_density = count(>(floor), deltas) runnera4_record_ram(game, frames, actions) runnerenv_step!(env, Int(actions[t])) jutariget_ram(env) jutaripairwise_corr(tape, cands) runnerglobal_structure(C, varying, tape) runnerScored against the exact intervention oracle Δy(u): run_intervention(checkpoint, actions, target_frame, horizon, cause).
The figure is generated from the committed record by
docs/gen_method_figures.py; the game frame
and each RAM cell's screen footprint are produced by
render_scenes.jl /
cell_footprints.jl.