cf. Jonas & Kording 2017
Granger causality infers direction from timing. The rule is simple: if the past of signal A helps predict the future of signal B beyond B's own past, then A is said to Granger-cause B. It is widely used in neuroscience and economics to draw directed influence graphs from recorded time series, without any intervention. We apply it to the VCS by treating the CPU, TIA, and RIOT activity as time series and asking which components predict which. The result is a directed graph of inferred influence that we compare to the true data-flow. The appeal is that it turns passive observation into a causal-looking claim. The deep flaw, on this machine, is that it equates 'happens earlier' with 'is the cause'. The VCS is clock-locked, so almost every signal is predictable from almost every other signal one cycle earlier, whether or not there is a real dependency. Granger causality therefore infers edges everywhere. Neither a longer lag nor a stricter threshold repairs this, because the shared clock creates predictability that is not causation. It is the cleanest demonstration in our battery that precedence is not cause, and that only an intervention can tell them apart.
Two adjacency matrices over the candidate RAM cells: the true data-flow graph and the graph this method recovered. A bright cell (row = cause, column = effect) is an edge; the difference between the two panels is the error. A6_cell_false_edge_rate_vs_true_dataflow = 1.000 — this example only (pong, state gameplay(seed=0,prefix=90)+15); the audit aggregate is below.
New to explainable AI? This block explains the method as it is used on neural networks, then how we run the very same method on an Atari 2600, where — unlike for a network — we know the right answer. Every implementation step links to the code. See also the XAI primer for the shared vocabulary.
You record two signals over time, say the activity of brain areas A and B. You cannot cut wires or stimulate anything. Can you still say which one drives the other? Clive Granger's idea (1969) is a clever proxy: a cause comes before its effect, and it carries information about the effect. So try to predict B's next value from B's own past. Then add A's past. If the prediction gets clearly better, A is said to Granger-cause B. Run this for every ordered pair and you get a directed graph of “who influences whom”, built purely from passive recordings.
The word “causality” is generous. What the test really measures is temporal precedence plus predictive information. That is not the same as a causal wire. If both A and B are driven by a hidden third signal C, and A sees C a moment earlier than B, then A's past predicts B perfectly, with no wire from A to B at all. This hidden common driver is a confounder. A clock-driven machine like the Atari is full of them. A6 measures how far the Granger graph sits from the true wiring, which on the VCS we can establish by direct intervention.
Neuroscience origin. Granger causality is a workhorse for “directed functional connectivity” in EEG, MEG, LFP and fMRI data (Seth et al. 2015). Jonas & Kording applied it to a simulated 6502 chip and found a graph that bore little resemblance to the real circuit.
Deep-learning analogue. In XAI it is temporal-precedence causal discovery on activation time series: record hidden units of a recurrent network, the residual stream across tokens, or the latent state of an RL agent over steps, then infer which units “drive” which.
The test. For an ordered pair \(x\to y\) and a lag \(p\), fit two linear autoregressions by least squares:
\[ \text{restricted:}\; y_t = a_0 + \sum_{l=1}^{p} a_l\,y_{t-l} + e_t, \qquad \text{unrestricted:}\; y_t = a_0 + \sum_{l=1}^{p} a_l\,y_{t-l} + \sum_{l=1}^{p} b_l\,x_{t-l} + e'_t . \]With residual sums of squares \(\mathrm{RSS}_r \ge \mathrm{RSS}_u\) over \(n\) usable steps, the Granger statistic and its F-test are
\[ G(x\to y) = \ln\frac{\mathrm{RSS}_r}{\mathrm{RSS}_u}, \qquad F = \frac{(\mathrm{RSS}_r-\mathrm{RSS}_u)/p}{\mathrm{RSS}_u/(n-2p-1)} . \]An edge \(x\to y\) is declared when the F-test's p-value falls below a significance level \(\alpha\) (usually 0.05).
Toy example. A hidden driver produces random numbers \(d = (3,1,4,1,5,9,2,6)\). Unit \(x\) copies it one step late, unit \(y\) two steps late: \(x_t = d_{t-1}\), \(y_t = d_{t-2}\). So \(x = (\cdot,3,1,4,1,5,9,2)\) and \(y = (\cdot,\cdot,3,1,4,1,5,9)\). Because \(d\) is random, \(y\)'s own past says little about its next value: \(\mathrm{RSS}_r\) is large. But \(y_t = x_{t-1}\) exactly, so adding \(x\)'s past drives \(\mathrm{RSS}_u\) to zero. \(G\) is huge, the p-value is tiny, and the test reports a strong edge \(x\to y\). There is no wire between them: both just read \(d\). Cutting \(x\) would leave \(y\) unchanged. Only that intervention reveals the truth.
| Recorded brain areas / hidden units | Two scopes: (a) the candidate RAM bytes, bytes the game is known to use for a score, a position, a timer, …; (b) three subsystems: the CPU's working RAM, the TIA (the video/sound chip, 64 registers) and the RIOT (timer and I/O chip). |
|---|---|
| An activity time series | Cell scope: the byte's value at every frame. Subsystem scope: how many bytes of that subsystem changed since the previous frame (its “firing rate”). |
| Directed functional connectivity | The Granger graph: an edge \(i\to j\) if byte \(i\)'s past significantly improves the prediction of byte \(j\) (lag 2, \(\alpha = 0.05\)). |
| The true anatomical wiring (unknown in a brain) | The intervention oracle's data-flow graph: edge \(i\to j\) if setting byte \(i\) to a new value at frame \(f^*\) and re-running the real ROM changes byte \(j\) 15 frames later. |
| A shared pacemaker / confounder | The frame loop. Every byte is updated in the same fixed order every frame, so many bytes move in lock-step without any data flowing between them. |
The runner A6_granger.jl builds the method's graph and the true graph on the same
nodes, then compares them.
a6_record_tape calls env_step! each frame and stores the
128 RAM bytes plus the 64 TIA registers.run_cell_granger tests every ordered pair of candidate
bytes with granger: lag \(p=2\), \(\alpha=0.05\), magnitude floor \(\tau=0\) (so
significance alone decides, the textbook rule). The least-squares fits and the F-distribution
p-value are written out by hand, so no statistics package is needed.subsystem_activity turns the tape into three
change-count series. The CPU series counts changes in RAM bytes 0x00–0x5F, the RIOT series in
0x60–0x7F, the TIA series in its register file. They are scored against a fixed truth: the CPU
writes the TIA, and the CPU reads the RIOT.true_dataflow_graph takes the checkpoint at \(f^*\) and, for each
candidate byte \(i\), pokes three values (0, base+17, base+37) with intervene_ram!. It
re-runs the 15-frame horizon and marks \(i\to j\) for every candidate byte \(j\) that ends up
different. A second graph from base+37 alone serves as a held-out probe.edge_rates gives the headline false-edge rate (share of Granger
edges that are not real), the missed-edge rate, and a direction disagreement: where both graphs
have one arrow between a pair, how often they point opposite ways. score_triad adds F (edge
F1 against the true graph), S (agreement with the held-out graph) and M (true edges divided by inferred
edges). A positive control scores the true graph against itself and must give F1 = 1.Differences from the textbook use. The true graph is a 15-frame reachability graph: an edge means byte \(j\) changed by any route, direct or indirect. It is measured at one state, while Granger is fitted on the whole trajectory. Both choices favour Granger, not penalise it. The bytes are also integers that jump in steps, not the smooth, noisy signals the linear model assumes.
XAI the interpretability method substrate the bit-exact emulator it runs on (jutari, Julia).
A6_granger.jl:431 — The bivariate Granger test: \(G=\ln(\mathrm{RSS}_r/\mathrm{RSS}_u)\), F statistic and p-value.A6_granger.jl:406 — Builds the lagged design matrices for the restricted and unrestricted fits.A6_granger.jl:611 — All ordered pairs of candidate bytes → the inferred directed graph.A6_granger.jl:559 — CPU / TIA / RIOT “firing rates”: bytes changed per frame.A6_granger.jl:646 — The ground-truth graph: poke each byte, re-run the ROM, see which bytes move.A6_granger.jl:678 — False-edge rate, missed-edge rate and direction disagreement.A6_granger.jl:839 — One game end to end: recording → both Granger scopes → true graph → triad → control.jutari_oracle.jl:176 — The intervention: write one value into one RAM byte of a copied machine.StellaEnvironment.jl:259 — One video frame of the real, bit-exact machine (recording and re-runs).oracle_intervene.jl:242 — The shared oracle that gates the gameplay state used by every Phase-A runner.The figure sets the Granger graph beside the true data-flow graph over the same candidate bytes (Pong). Here the two hardly overlap: the true graph has whole rows of edges (the score and ball bytes reach many cells within 15 frames), while Granger asserts almost nothing, and what it does assert is not in the truth. In games with many moving bytes (Ms. Pac-Man, Q*bert) the Granger graph turns dense instead, with many bytes appearing to drive many others. Either way most Granger edges have no counterpart in the truth, which is why the false-edge rate sits near its ceiling and the faithfulness is among the lowest in the battery. The subsystem scope in the record tells the same story at coarse scale. Changing the lag or the threshold moves edges around but does not fix the picture, because the shared frame loop creates the predictability.
julia --project=jutari tools/xai_study/phaseA_kording/A6_granger.jl --games core
Writes A6_<game>.json/.npz (Granger statistics, p-values, inferred and true adjacency matrices, edge rates, triad) into tools/xai_study/phaseA_kording/out/. Try --lag 4 or --alpha 0.01 to see that the false edges persist. Minutes per game; the true graph costs three ROM re-runs per candidate byte.
Read next: A1 · Connectomics / data-flow graph · A4 · Pairwise correlations · A2 · Single-unit lesions · Activation patching / causal mediation
The score is the false-edge rate of the Granger-inferred graph against the true data-flow. We let Granger causality infer directed edges from the timing of CPU, TIA, and RIOT activity, then compare each inferred edge to the true wiring the oracle certifies. An inferred edge that is not real is a false edge, and the false-edge rate counts them; faithfulness is one minus that rate. On the VCS the rate is at or near one, because the machine is clock-locked and almost every signal is predictable from almost every other one cycle earlier. So Granger causality infers edges everywhere, and nearly all of them are spurious. Neither a longer lag nor a stricter threshold repairs this, because the shared clock, not a real dependency, creates the predictability. The grading is always against the intervention oracle, which is the only thing that separates precedence from causation here. There is no sprite-position hard-case for this method; its output is a directed graph. The audit box reports the measured faithfulness across the scored games, and it is among the lowest in the battery, which is the point: timing-based causal inference fails on a machine whose timing is shared.
The score is measured against the §1 intervention oracle — never against another interpretability method. F (faithful) is always vs the oracle; see the execution stack. How each of F / S / M is actually computed for this method (and whether it matches the paper) is in the box just below; the numbers are in the In the audit box under it.
The exact formula this method uses for each score, read from its runner, and
whether it matches the paper's §3 (F ∧ S ∧ M triad) definition.
From the committed audit
fsm_math_phaseA.json.
| Axis | Formula (as computed) | What it measures | Matches §3? |
|---|---|---|---|
| F faithfulness | \(F = F_1^{\mathrm{edge}}(\hat{E}, E^{\star})\) | F1 of the Granger-inferred directed edges against the exact-oracle true data-flow graph over 8 candidate cells. | ✓ matches Graph-valued primitive = edge F1 vs true data-flow graph, matching the paper. Runner A6_granger.jl,; record out/A6_pong.json triad.F=0.0. how it's measured — call stack:
|
| S sufficiency | \(S = \dfrac{\#\{(i,j):\ \hat{E}_{ij}=E^{\star,\mathrm{held}}_{ij}\}}{\#\,\mathrm{off\text{-}diag}} \in [0,1]\) | Directed off-diagonal edge agreement (accuracy) between the Granger graph and the held-out base+37 true data-flow. | ✓ matches Directed off-diagonal edge agreement between the Granger graph and the held-out base+37 true data-flow: a non-null held-out sufficiency, the [0,1] fraction-within-tolerance special case §3 (sec:triad) admits. A6_granger.jl; record S=0.821. how it's measured — call stack:
|
| M minimality | \(M = |U^{\star}| / |\hat{U}| = |E^{\star}| / |\hat{E}| \in (0,1]\) | True-graph directed-edge count over the number of Granger-inferred edges (edge-level |U*|/|U_hat|). | ✓ matches Runner now emits M = |U*|/|U_hat| over edges (M_note '|true edges|=12 / |discovered edges|=1'), the paper's minimality ratio at the edge level. Record out/A6_pong.json extra.triad.M=1.0. Replaced the former 1 − false-edge rate (edge precision). how it's measured — call stack:
|
✓ matches = the same quantity as §3; ◐ partial = the same kind of estimator but a differing detail; ✗ does not match = a different quantity (see the note). Definitions: F = agreement with the oracle's true causal effects Δy(u); S = held-out predictive score in [−1, 1]; M = |U*|/|Û| (true-minimal-set size / named-set size).
This is the method's entry in the actual cross-method audit — scored on the paper's correctness triad, each axis a mean over 38 of the 44 scored games (38 committed §R records), not the single example shown above. Tradition: correlational. The example figure (Pong) is one of those records.
F faithfulness (scored vs the oracle for every method) · S sufficiency (held-out predictive score in [−1, 1]; a negative value means the explanation predicts held-out interventions worse than the unperturbed baseline; reported for the predictive methods across all three phases where the calibration/held-out split is defined) · M minimality (true-minimal-set / named-set; where the method names a cause set) — n/a otherwise, per the paper's F ∧ S ∧ M triad.
Source: leaderboard.json · the whole leaderboard is on the
methods page and the
Paper 2 audit.
This method's faithfulness on each of the 38 scored games (all-regime F; this method has no content-vs-position split (see the note below)). Click a header to
sort. Every number is read from site_data.json
(methods.A6_granger.per_game).
content-F and position-F are n/a for this method. The content-vs-position split applies only to pixel-attribution methods — those that produce an attribution map over the framebuffer, whose score can be read separately on the smooth content outputs and the discrete position outputs (where the naive gradient is provably zero). This method does not attribute to a pixel output; it is graded by its own fidelity metric (A6_cell_false_edge_rate_vs_true_dataflow), reported as the F column, so there is no content or position regime to separate.
M is n/a on 3 of these games. Minimality is M = |U*|/|Û|, the size of the true minimal cause set over the size of the set the method actually names. On a game where the method names nothing (|Û| = 0 — it discovers no circuit, its attribution map is all-zero, or it decodes no cell), that ratio is undefined, so M is left blank rather than scored. The aggregate M is the mean over the 35 games where the method did name a cause set.
| Game | F (all) | content-F | position-F | S | M | Note |
|---|---|---|---|---|---|---|
| Video Pinball | 0.500 | n/a | n/a | 0.952 | 1.000 | |
| Enduro | 0.400 | n/a | n/a | 0.500 | 1.000 | |
| Frostbite | 0.345 | n/a | n/a | 0.901 | 0.812 | |
| Yars' Revenge | 0.345 | n/a | n/a | 0.878 | 0.450 | |
| Venture | 0.344 | n/a | n/a | 0.709 | 0.488 | |
| Atlantis | 0.333 | n/a | n/a | 0.944 | 0.500 | |
| Phoenix | 0.333 | n/a | n/a | 0.970 | 0.200 | |
| Skiing | 0.333 | n/a | n/a | 0.889 | 1.000 | |
| Ice Hockey | 0.323 | n/a | n/a | 0.811 | 0.550 | |
| Double Dunk | 0.298 | n/a | n/a | 0.742 | 0.270 | |
| Bowling | 0.286 | n/a | n/a | 0.992 | 1.000 | |
| Chopper Command | 0.264 | n/a | n/a | 0.910 | 0.828 | |
| Demon Attack | 0.256 | n/a | n/a | 0.876 | 0.857 | |
| Berzerk | 0.255 | n/a | n/a | 0.881 | 0.719 | |
| Montezuma's Revenge | 0.210 | n/a | n/a | 0.962 | 1.000 | |
| Breakout | 0.200 | n/a | n/a | 0.931 | 1.000 | |
| Centipede | 0.143 | n/a | n/a | 0.847 | 0.273 | |
| Carnival | 0.118 | n/a | n/a | 0.974 | 1.000 | |
| Assault | 0.111 | n/a | n/a | 0.976 | 1.000 | |
| Air Raid | 0.105 | n/a | n/a | 0.891 | 0.727 | |
| Private Eye | 0.100 | n/a | n/a | 0.980 | 0.818 | |
| Tennis | 0.089 | n/a | n/a | 0.805 | 0.324 | |
| Boxing | 0.083 | n/a | n/a | 0.818 | 1.000 | |
| Riverraid | 0.074 | n/a | n/a | 0.896 | 0.636 | |
| Seaquest | 0.061 | n/a | n/a | 0.947 | 0.737 | |
| Road Runner | 0.037 | n/a | n/a | 0.660 | 0.069 | no true-cause signal |
| Q*bert | 0.012 | n/a | n/a | 0.560 | 0.006 | no true-cause signal |
| Alien | 0.000 | n/a | n/a | 1.000 | n/a | no true-cause signal |
| Asterix | 0.000 | n/a | n/a | 0.917 | 1.000 | no true-cause signal |
| Hero | 0.000 | n/a | n/a | 0.982 | 1.000 | no true-cause signal |
| Jamesbond | 0.000 | n/a | n/a | 0.853 | 0.150 | no true-cause signal |
| Kangaroo | 0.000 | n/a | n/a | 0.995 | n/a | no true-cause signal |
| Krull | 0.000 | n/a | n/a | 0.985 | 1.000 | no true-cause signal |
| Ms. Pac-Man | 0.000 | n/a | n/a | 0.746 | 0.261 | no true-cause signal |
| Pacman | 0.000 | n/a | n/a | 0.982 | n/a | no true-cause signal |
| Pitfall | 0.000 | n/a | n/a | 0.912 | 0.846 | no true-cause signal |
| Pong | 0.000 | n/a | n/a | 0.821 | 1.000 | no true-cause signal |
| Space Invaders | 0.000 | n/a | n/a | 0.956 | 1.000 | no true-cause signal |
The path from this method's runner (compute_game) into the bit-exact VCS substrate, and the
computation it involves. Every step links to the exact source on main. From
callstack_phaseA.json.
build_a6_shared_state(game) -> build_shared_testbed(game; ...) runnerfunction build_shared_testbed(game; ...) runnercause_density = count(>(floor), deltas) runnera6_record_tape(game, frames, actions) runnerenv_step!(env, Int(actions[t])) jutariget_ram(env) jutarirun_cell_granger(tape, cands; p, granger_tau, alpha) runnerrun_subsystem_granger(tape; p, granger_tau, ...) runnerScored against the exact intervention oracle Δy(u): run_intervention(checkpoint, actions, target_frame, horizon, cause).
The figure is generated from the committed record by
docs/gen_method_figures.py; the game frame
and each RAM cell's screen footprint are produced by
render_scenes.jl /
cell_footprints.jl.