cf. Jonas & Kording 2017
Whole-state recording is the simplest possible method: write down everything. At each step we record every bit of RAM and every register, the complete internal state over time. This is the raw material that every other method in the study works from. In neuroscience it is the dream of recording every neuron at once, and it is often treated as the thing that would finally let us understand the system. The intuition is that if we have the full state, we have hidden nothing, so we must have the explanation. We include it as a deliberate baseline to test that intuition. By construction the complete record is perfectly faithful and perfectly sufficient: it contains every cause, so nothing is missing, and it can predict any intervention because it holds the whole machine. The point of including it is to show what such a record still lacks. A dump of all 128 RAM cells is not an account of the computation. It names everything, so it names nothing in particular, and the true cause of any given output is a tiny handful of those cells. Completeness is not understanding, and this method makes that concrete.
Left: the frame. Right: this method's per-cell/per-component result against the ground-truth importance, with the RAM cells labelled. wholestate_minimality_M_vs_oracle = 0.031 — this example only (pong, state gameplay(seed=0,prefix=90)+15); the audit aggregate is below.
The score that matters for this method is minimality, not faithfulness. Recording the whole state is perfectly faithful, because it contains every cause, and perfectly sufficient, because it holds the whole machine and can predict any intervention. So F and S are both at the ceiling by construction. Minimality is defined as the size of the true minimal cause set divided by the size of the set the explanation names. The whole-state dump names all 128 RAM cells, while the true cause of any output is a handful of them, so the ratio is tiny and the minimality is very low. That is the whole point of including this baseline: a complete record is faithful and sufficient yet says nothing in particular, because it withholds nothing. The grading is always against the intervention oracle, which supplies the true minimal set. There is no sprite-position hard-case here; the method makes no positional claim. The audit box reports the exact per-axis triad, F, S, and M, read straight from the leaderboard, so you can see faithfulness and sufficiency at the ceiling beside a minimality near the floor.
The score is measured against the §1 intervention oracle — never against another interpretability method. F (faithful) is always vs the oracle; see the execution stack. How each of F / S / M is actually computed for this method (and whether it matches the paper) is in the box just below; the numbers are in the In the audit box under it.
The exact formula this method uses for each score, read from its runner, and
whether it matches the paper's §3 (F ∧ S ∧ M triad) definition.
From the committed audit
fsm_math_phaseA.json.
| Axis | Formula (as computed) | What it measures | Matches §3? |
|---|---|---|---|
| F faithfulness | \(F = \mathrm{recall}\!\left(U^{\star}\mid \hat{U}=\text{whole state}\right) = 1\) | Recall of the oracle-causal cells under the record-everything map; trivially 1.0 because the whole state includes every cell. | ◐ partial Scored vs the oracle causal cells but the primitive is recall (not the correlation/edge-F1 the paper specifies); it is a hard-coded 1.0 by construction, consistent with the paper's 'trivially faithful' whole-state baseline. Runner A8_wholestate.jl,; record out/A8_pong.json triad.F=1.0. how it's measured — call stack:
|
| S sufficiency | \(S = \mathbb{1}\!\left[\text{restored 128-byte RAM} = \text{exact RAM + frame}\right] \in \{0,1\}\) | 1 if the recorded whole 128-byte RAM state, restored, reproduces the exact RAM and frame (lossless), else 0. | ✓ matches Literal sufficiency: the recorded whole 128-byte RAM state, restored, reproduces the exact RAM and frame (bit-exact re-run). §3 (sec:triad) admits the [0,1] fraction-within-tolerance special case of the held-out test; this boolean 1/0 hit is that special case at its extreme. A8_wholestate.jl; record S=1.0. how it's measured — call stack:
|
| M minimality | \(M = |U^{\star}| / |\hat{U}| = n_{\mathrm{causal}} / 128\) | Number of oracle-causal cells over the size of the named set (128), the whole-state minimality floor. | ✓ matches The only Phase-A method whose M is exactly |U*|/|U_hat|; paper cites whole-state M=0.07. Headline metric_name = wholestate_minimality_M_vs_oracle. A8_wholestate.jl,; record out/A8_pong.json M=0.03125 = 4/128. how it's measured — call stack:
|
✓ matches = the same quantity as §3; ◐ partial = the same kind of estimator but a differing detail; ✗ does not match = a different quantity (see the note). Definitions: F = agreement with the oracle's true causal effects Δy(u); S = held-out predictive score in [−1, 1]; M = |U*|/|Û| (true-minimal-set size / named-set size).
This is the method's entry in the actual cross-method audit — scored on the paper's correctness triad, each axis a mean over all 42 scored games (42 committed §R records), not the single example shown above. Tradition: descriptive. The example figure (Pong) is one of those records.
F faithfulness (scored vs the oracle for every method) · S sufficiency (held-out predictive score in [−1, 1]; a negative value means the explanation predicts held-out interventions worse than the unperturbed baseline; reported for the predictive methods across all three phases where the calibration/held-out split is defined) · M minimality (true-minimal-set / named-set; where the method names a cause set) — n/a otherwise, per the paper's F ∧ S ∧ M triad.
Source: leaderboard.json · the whole leaderboard is on the
methods page and the
Paper 2 audit.
This method's faithfulness on each of the 42 scored games (all-regime F; this method has no content-vs-position split (see the note below)). Click a header to
sort. Every number is read from site_data.json
(methods.A8_wholestate.per_game).
content-F and position-F are n/a for this method. The content-vs-position split applies only to pixel-attribution methods — those that produce an attribution map over the framebuffer, whose score can be read separately on the smooth content outputs and the discrete position outputs (where the naive gradient is provably zero). This method does not attribute to a pixel output; it is graded by its own fidelity metric (wholestate_minimality_M_vs_oracle), reported as the F column, so there is no content or position regime to separate.
| Game | F (all) | content-F | position-F | S | M | Note |
|---|---|---|---|---|---|---|
| Air Raid | 1.000 | n/a | n/a | 1.000 | 0.078 | recovers the true causes |
| Alien | 1.000 | n/a | n/a | 1.000 | 0.031 | recovers the true causes |
| Assault | 1.000 | n/a | n/a | 1.000 | 0.070 | recovers the true causes |
| Atlantis | 1.000 | n/a | n/a | 1.000 | 0.055 | recovers the true causes |
| Bank Heist | 1.000 | n/a | n/a | 1.000 | 0.031 | recovers the true causes |
| Beam Rider | 1.000 | n/a | n/a | 1.000 | 0.016 | recovers the true causes |
| Berzerk | 1.000 | n/a | n/a | 1.000 | 0.109 | recovers the true causes |
| Bowling | 1.000 | n/a | n/a | 1.000 | 0.062 | recovers the true causes |
| Boxing | 1.000 | n/a | n/a | 1.000 | 0.070 | recovers the true causes |
| Breakout | 1.000 | n/a | n/a | 1.000 | 0.055 | recovers the true causes |
| Carnival | 1.000 | n/a | n/a | 1.000 | 0.117 | recovers the true causes |
| Centipede | 1.000 | n/a | n/a | 1.000 | 0.055 | recovers the true causes |
| Chopper Command | 1.000 | n/a | n/a | 1.000 | 0.078 | recovers the true causes |
| Demon Attack | 1.000 | n/a | n/a | 1.000 | 0.070 | recovers the true causes |
| Double Dunk | 1.000 | n/a | n/a | 1.000 | 0.094 | recovers the true causes |
| Fishing Derby | 1.000 | n/a | n/a | 1.000 | 0.039 | recovers the true causes |
| Freeway | 1.000 | n/a | n/a | 1.000 | 0.109 | recovers the true causes |
| Frostbite | 1.000 | n/a | n/a | 1.000 | 0.109 | recovers the true causes |
| Gopher | 1.000 | n/a | n/a | 1.000 | 0.148 | recovers the true causes |
| Hero | 1.000 | n/a | n/a | 1.000 | 0.086 | recovers the true causes |
| Ice Hockey | 1.000 | n/a | n/a | 1.000 | 0.070 | recovers the true causes |
| Jamesbond | 1.000 | n/a | n/a | 1.000 | 0.055 | recovers the true causes |
| Kangaroo | 1.000 | n/a | n/a | 1.000 | 0.078 | recovers the true causes |
| Krull | 1.000 | n/a | n/a | 1.000 | 0.086 | recovers the true causes |
| Kung-Fu Master | 1.000 | n/a | n/a | 1.000 | 0.047 | recovers the true causes |
| Montezuma's Revenge | 1.000 | n/a | n/a | 1.000 | 0.102 | recovers the true causes |
| Ms. Pac-Man | 1.000 | n/a | n/a | 1.000 | 0.109 | recovers the true causes |
| Name This Game | 1.000 | n/a | n/a | 1.000 | 0.031 | recovers the true causes |
| Pacman | 1.000 | n/a | n/a | 1.000 | 0.055 | recovers the true causes |
| Phoenix | 1.000 | n/a | n/a | 1.000 | 0.062 | recovers the true causes |
| Pitfall | 1.000 | n/a | n/a | 1.000 | 0.086 | recovers the true causes |
| Pong | 1.000 | n/a | n/a | 1.000 | 0.031 | recovers the true causes |
| Private Eye | 1.000 | n/a | n/a | 1.000 | 0.094 | recovers the true causes |
| Q*bert | 1.000 | n/a | n/a | 1.000 | 0.055 | recovers the true causes |
| Riverraid | 1.000 | n/a | n/a | 1.000 | 0.148 | recovers the true causes |
| Road Runner | 1.000 | n/a | n/a | 1.000 | 0.047 | recovers the true causes |
| Seaquest | 1.000 | n/a | n/a | 1.000 | 0.117 | recovers the true causes |
| Space Invaders | 1.000 | n/a | n/a | 1.000 | 0.094 | recovers the true causes |
| Tennis | 1.000 | n/a | n/a | 1.000 | 0.086 | recovers the true causes |
| Venture | 1.000 | n/a | n/a | 1.000 | 0.125 | recovers the true causes |
| Video Pinball | 1.000 | n/a | n/a | 1.000 | 0.055 | recovers the true causes |
| Yars' Revenge | 1.000 | n/a | n/a | 1.000 | 0.055 | recovers the true causes |
The path from this method's runner (run_game) into the bit-exact VCS substrate, and the
computation it involves. Every step links to the exact source on main. From
callstack_phaseA.json.
build_a8_shared_state(game) -> build_shared_testbed(game; ...) runnerfunction build_shared_testbed(game; ...) runnercause_density = count(>(floor), deltas) runnerboot_replay(game, actions, target_frame) runnerrecord_whole_state(game, actions) runnerenv_step!(env, Int(actions[t])) jutarirestore_whole_ram(checkpoint, ram) runnerverify_wholestate_sufficiency(checkpoint, at_target) runnerverify_minimal_sufficiency(checkpoint, at_target, causal_idx) runneroracle_causal_importance(checkpoint, tail, cands, at_target) runnerScored against the exact intervention oracle Δy(u): run_intervention(checkpoint, actions, target_frame, horizon, cause).
The figure is generated from the committed record by
docs/gen_method_figures.py; the game frame
and each RAM cell's screen footprint are produced by
render_scenes.jl /
cell_footprints.jl.