cf. Jonas & Kording 2017
A tuning curve measures how strongly a unit responds to some variable. In neuroscience one plots a neuron's firing rate against, say, the angle of a bar, and a sharp peak is read as the neuron being 'tuned' to that feature. We build the same curves for RAM cells. For each cell we plot its value against a game variable and against screen luminance, then measure how strongly the cell tracks each one. A cell that closely follows a game variable is called tuned to it. The intuition is that a strongly tuned cell must encode that variable and therefore drive the behaviour. This is one of the workhorses of systems neuroscience. On the VCS it is actively misleading. Many cells co-vary with the beam clock and the frame counter, so apparent tuning is cheap and common. A cell can track a game variable perfectly while playing no causal role in producing it. The method thus flags cells that look meaningful but are not, which is exactly the trap we set out to measure. It is the same present-versus-used problem that also defeats linear probing.
Each point is a RAM cell: its true causal importance (x) against its tuning strength to a game variable (y). Red points are strongly tuned yet not causal — the trap the metric measures. A3_game_variable_spurious_tuning_rate = 0.500 — this example only (pong, state gameplay(seed=0,prefix=90)+15); the audit aggregate is below.
The score is a spurious-tuning rate: the fraction of cells that are strongly tuned to a game variable yet are not among the oracle's causal cells. We first find the cells whose value tracks a game variable strongly, then ask the intervention oracle which of those cells actually cause the output. A tuned cell that is not causal is a false positive, and the spurious-tuning rate counts them. Faithfulness is one minus that rate, so a method that only flags true causes scores high and a method that flags many non-causal cells scores low. On the VCS the rate is high, because many cells co-vary with the beam clock, so tuning is cheap and often unrelated to causation. This is the same present-versus-used trap that also defeats linear probing: a cell can carry a variable without the program using it. The grading is always against the oracle. There is no sprite-position hard-case here, because the output is a tuning judgement, not a coordinate. Because per-game correlations for this family can go negative, the aggregate clips each per-game value at zero before averaging, so a method with no real signal lands near zero rather than at a spurious floor. The audit box reports the aggregated faithfulness.
The score is measured against the §1 intervention oracle — never against another interpretability method. F (faithful) is always vs the oracle; see the execution stack. How each of F / S / M is actually computed for this method (and whether it matches the paper) is in the box just below; the numbers are in the In the audit box under it.
The exact formula this method uses for each score, read from its runner, and
whether it matches the paper's §3 (F ∧ S ∧ M triad) definition.
From the committed audit
fsm_math_phaseA.json.
| Axis | Formula (as computed) | What it measures | Matches §3? |
|---|---|---|---|
| F faithfulness | \(F = \rho_{\mathrm{Spearman}}\!\left(\mathrm{attr},\ \Delta y(u)\right)\) | Spearman correlation between per-cell game-variable tuning selectivity and the oracle causal importance over 8 candidate cells. | ✓ matches Scored vs oracle causal importance (not vs external labels); the per-game clip-at-0-then-mean is applied at aggregation (paper cites A3 = 0.213 clip-at-zero mean). Runner A3_tuning.jl,; record out/A3_tuning_pong.json triad.F=-0.007. how it's measured — call stack:
|
| S sufficiency | \(S = \rho_{\mathrm{Pearson}}\!\left(\mathrm{attr},\ \Delta y_{\mathrm{held\text{-}out}}(u)\right) \in [-1,1]\) | Pearson correlation of the selectivity map vs the held-out do(base+37) screen break. | ✓ matches Held-out predictive correlation in [-1,1] of the selectivity map vs the held-out do(base+37) screen break: the §3 (sec:triad) form. A3_tuning.jl; record S=0.441 (battery air_raid). how it's measured — call stack:
|
| M minimality | \(M = |U^{\star}| / |\hat{U}| \in (0,1]\quad U^{\star}=\{u:\Delta y(u)>0\},\ \hat{U}=\{u:|\mathrm{attr}(u)|>\tau\}\) | Number of oracle-causal cells over the number of strongly-tuned cells the method names. | ✓ matches Runner now emits M = |U*|/|U_hat| (M_note '|U*|=10 (oracle movers) / |U_hat|=7 (method above-threshold)'), the paper's set-size ratio. Record out/A3_tuning_*.json extra.triad.M. Replaced the former 1 − spurious-tuning rate. how it's measured — call stack:
|
✓ matches = the same quantity as §3; ◐ partial = the same kind of estimator but a differing detail; ✗ does not match = a different quantity (see the note). Definitions: F = agreement with the oracle's true causal effects Δy(u); S = held-out predictive score in [−1, 1]; M = |U*|/|Û| (true-minimal-set size / named-set size).
This is the method's entry in the actual cross-method audit — scored on the paper's correctness triad, each axis a mean over all 42 scored games (42 committed §R records), not the single example shown above. Tradition: correlational. The example figure (Pong) is one of those records.
F faithfulness (scored vs the oracle for every method) · S sufficiency (held-out predictive score in [−1, 1]; a negative value means the explanation predicts held-out interventions worse than the unperturbed baseline; reported for the predictive methods across all three phases where the calibration/held-out split is defined) · M minimality (true-minimal-set / named-set; where the method names a cause set) — n/a otherwise, per the paper's F ∧ S ∧ M triad.
Source: leaderboard.json · the whole leaderboard is on the
methods page and the
Paper 2 audit.
This method's faithfulness on each of the 42 scored games (all-regime F; this method has no content-vs-position split (see the note below)). Click a header to
sort. Every number is read from site_data.json
(methods.A3_tuning.per_game).
content-F and position-F are n/a for this method. The content-vs-position split applies only to pixel-attribution methods — those that produce an attribution map over the framebuffer, whose score can be read separately on the smooth content outputs and the discrete position outputs (where the naive gradient is provably zero). This method does not attribute to a pixel output; it is graded by its own fidelity metric (A3_game_variable_spurious_tuning_rate), reported as the F column, so there is no content or position regime to separate.
M is n/a on 5 of these games. Minimality is M = |U*|/|Û|, the size of the true minimal cause set over the size of the set the method actually names. On a game where the method names nothing (|Û| = 0 — it discovers no circuit, its attribution map is all-zero, or it decodes no cell), that ratio is undefined, so M is left blank rather than scored. The aggregate M is the mean over the 37 games where the method did name a cause set.
S is negative on 13 of these games. Sufficiency asks whether the explanation can predict the outcome of held-out interventions it was not fit on; for this method S is the held-out predictive correlation between its claim and the true effect, so it lies in [−1, 1]. A value near 0 means the explanation predicts no better than the unperturbed output; a negative value means it predicts worse — its held-out claim is anti-correlated with the truth (named yet causally unused). The aggregate S is the mean over the scored games.
| Game | F (all) | content-F | position-F | S | M | Note |
|---|---|---|---|---|---|---|
| Name This Game | 0.866 | n/a | n/a | 0.754 | 1.000 | recovers the true causes |
| Freeway | 0.829 | n/a | n/a | 0.873 | 1.000 | |
| Boxing | 0.681 | n/a | n/a | 0.728 | 1.000 | |
| Double Dunk | 0.617 | n/a | n/a | 0.808 | 1.000 | |
| Bowling | 0.603 | n/a | n/a | 0.573 | 1.000 | |
| Bank Heist | 0.564 | n/a | n/a | 0.247 | 1.000 | |
| Berzerk | 0.513 | n/a | n/a | 0.223 | 0.875 | |
| Fishing Derby | 0.491 | n/a | n/a | 0.490 | 0.714 | |
| Frostbite | 0.476 | n/a | n/a | 0.346 | 1.000 | |
| Q*bert | 0.467 | n/a | n/a | 0.183 | 0.467 | |
| Air Raid | 0.445 | n/a | n/a | 0.441 | 1.000 | |
| Demon Attack | 0.406 | n/a | n/a | 0.200 | 0.750 | |
| Yars' Revenge | 0.395 | n/a | n/a | 0.223 | 0.700 | |
| Krull | 0.305 | n/a | n/a | 0.184 | 1.000 | |
| Tennis | 0.231 | n/a | n/a | -0.217 | 1.000 | |
| Hero | 0.229 | n/a | n/a | -0.114 | 1.000 | |
| Private Eye | 0.223 | n/a | n/a | 0.327 | 1.000 | |
| Jamesbond | 0.188 | n/a | n/a | 0.130 | 0.875 | |
| Pitfall | 0.146 | n/a | n/a | 0.258 | 0.846 | |
| Alien | 0.105 | n/a | n/a | 0.245 | 1.000 | |
| Seaquest | 0.078 | n/a | n/a | 0.311 | 1.000 | |
| Assault | 0.051 | n/a | n/a | -0.004 | 0.900 | |
| Phoenix | 0.025 | n/a | n/a | -0.184 | 1.000 | no true-cause signal |
| Chopper Command | 0.007 | n/a | n/a | 0.180 | 0.769 | no true-cause signal |
| Atlantis | 0.000 | n/a | n/a | -0.446 | 1.000 | no true-cause signal |
| Beam Rider | 0.000 | n/a | n/a | 0.000 | n/a | no true-cause signal |
| Breakout | 0.000 | n/a | n/a | -0.381 | 1.000 | no true-cause signal |
| Carnival | 0.000 | n/a | n/a | -0.028 | 1.000 | no true-cause signal |
| Centipede | 0.000 | n/a | n/a | -0.589 | 1.000 | no true-cause signal |
| Gopher | 0.000 | n/a | n/a | 0.000 | n/a | no true-cause signal |
| Ice Hockey | 0.000 | n/a | n/a | 0.032 | 1.000 | no true-cause signal |
| Kangaroo | 0.000 | n/a | n/a | 0.000 | n/a | no true-cause signal |
| Kung-Fu Master | 0.000 | n/a | n/a | 0.000 | n/a | no true-cause signal |
| Montezuma's Revenge | 0.000 | n/a | n/a | -0.226 | 1.000 | no true-cause signal |
| Ms. Pac-Man | 0.000 | n/a | n/a | 0.000 | n/a | no true-cause signal |
| Pacman | 0.000 | n/a | n/a | -0.304 | 1.000 | no true-cause signal |
| Pong | 0.000 | n/a | n/a | -0.412 | 1.000 | no true-cause signal |
| Riverraid | 0.000 | n/a | n/a | 0.092 | 1.000 | no true-cause signal |
| Road Runner | 0.000 | n/a | n/a | 0.083 | 0.429 | no true-cause signal |
| Space Invaders | 0.000 | n/a | n/a | -0.070 | 1.000 | no true-cause signal |
| Venture | 0.000 | n/a | n/a | -0.083 | 1.000 | no true-cause signal |
| Video Pinball | 0.000 | n/a | n/a | 0.472 | 1.000 | no true-cause signal |
The path from this method's runner (compute_game) into the bit-exact VCS substrate, and the
computation it involves. Every step links to the exact source on main. From
callstack_phaseA.json.
build_a3_shared_state(game) -> build_shared_testbed(game; ...) runnerfunction build_shared_testbed(game; ...) runnercause_density = count(>(floor), deltas) runnerrecord_parity_trajectory(game, actions) runnerenv_step!(env, Int(actions[t])) jutariget_ram(env) jutarigame_variable_signals(traj, cands) runnerluminance_and_clock(traj) runnerrun_a3_tuning(traj, cands, oracle_importance; tau=0.7) runnerScored against the exact intervention oracle Δy(u): run_intervention(checkpoint, actions, target_frame, horizon, cause).
The figure is generated from the committed record by
docs/gen_method_figures.py; the game frame
and each RAM cell's screen footprint are produced by
render_scenes.jl /
cell_footprints.jl.